From c6e15b2056517fc3be84ee700b83633af988122a Mon Sep 17 00:00:00 2001 From: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Date: Tue, 3 Dec 2024 18:04:08 +0200 Subject: [PATCH] Docs: `INSTANT` DDL and `RANGE PARTITIONING` support for managed schema migrations (#1899) * document '--prefer-instant-ddl' DDL strategy flag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * clarify removal of old table format Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix ordering of schema change pages Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * preemptively add link to instant DDL page Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * preemptively add more link to instant DDL page Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Adding INSTANT DDL docs for v22 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * INSTANT DDL in advanced usage Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * formatting Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add notes and exceptions Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * partitioning notes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * instant DDL docs for older versions Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * typo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix excessive text Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove /8.0/ from urls Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> --------- Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> --- .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/concurrent-migrations.md | 2 +- .../schema-changes/ddl-strategies.md | 5 ++ .../schema-changes/ddl-strategy-flags.md | 2 + .../schema-changes/instant-ddl-migrations.md | 88 +++++++++++++++++++ .../managed-online-schema-changes.md | 1 + .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/concurrent-migrations.md | 2 +- .../schema-changes/ddl-strategies.md | 5 ++ .../schema-changes/ddl-strategy-flags.md | 2 + .../schema-changes/instant-ddl-migrations.md | 88 +++++++++++++++++++ .../managed-online-schema-changes.md | 1 + .../schema-changes/validating-migrations.md | 2 +- .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/concurrent-migrations.md | 2 +- .../schema-changes/ddl-strategies.md | 5 ++ .../schema-changes/ddl-strategy-flags.md | 2 + .../schema-changes/instant-ddl-migrations.md | 88 +++++++++++++++++++ .../managed-online-schema-changes.md | 1 + .../schema-changes/validating-migrations.md | 2 +- .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/concurrent-migrations.md | 2 +- .../schema-changes/ddl-strategies.md | 5 ++ .../schema-changes/ddl-strategy-flags.md | 2 + .../schema-changes/instant-ddl-migrations.md | 88 +++++++++++++++++++ .../managed-online-schema-changes.md | 1 + .../schema-changes/table-lifecycle.md | 2 +- .../schema-changes/validating-migrations.md | 2 +- .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/ddl-strategy-flags.md | 2 + .../managed-online-schema-changes.md | 1 + .../reference/features/schema-management.md | 1 + .../schema-changes/advanced-usage.md | 11 +++ .../schema-changes/concurrent-migrations.md | 2 +- .../schema-changes/ddl-strategy-flags.md | 2 + .../managed-online-schema-changes.md | 1 + 41 files changed, 471 insertions(+), 9 deletions(-) create mode 100644 content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md create mode 100644 content/en/docs/20.0/user-guides/schema-changes/instant-ddl-migrations.md create mode 100644 content/en/docs/21.0/user-guides/schema-changes/instant-ddl-migrations.md create mode 100644 content/en/docs/22.0/user-guides/schema-changes/instant-ddl-migrations.md diff --git a/content/en/docs/19.0/reference/features/schema-management.md b/content/en/docs/19.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/19.0/reference/features/schema-management.md +++ b/content/en/docs/19.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md index 31abdc054..856cbce51 100644 --- a/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md @@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order- - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md index aabf08cf1..4984aaddd 100644 --- a/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md @@ -1,6 +1,6 @@ --- title: Concurrent migration execution -weight: 15 +weight: 16 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/'] --- diff --git a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md index 72f08204a..788c0f8d0 100644 --- a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md +++ b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md @@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic. - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table. +#### Notes and exceptions + +- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible. +- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes) + ### gh-ost [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool. diff --git a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md index 29207c877..69ca8d418 100644 --- a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -39,6 +39,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--retain-artifacts=`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value. - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. diff --git a/content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md b/content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md new file mode 100644 index 000000000..3ed7a37cd --- /dev/null +++ b/content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md @@ -0,0 +1,88 @@ +--- +title: INSTANT DDL migrations +weight: 15 +aliases: ['/docs/user-guides/schema-changes/instant-ddl-migrations/'] +--- + +MySQL's [INSTANT DDL](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) is accessible in both unmanaged and managed vitess migrations. + +`INSTANT` DDL refers to `ALTER TABLE ... ALGORITHM=INSTANT` which runs near-instantaneously, though some locking is still witnessed in heavy workloads. It is limited to a subset of operations. The [documentation](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) lists the general limitations but does not cover the precise range of scenarios. + +A common naive approach to using `INSTANT` DDL is to optimistically attempt an `ALTER TABLE ... ALGORITHM=INSTANT`, and if that turns to be unsupported by `INSTANT` DDL, resort to standard `ALTER TABLE`. + +## INSTANT DDL in unmanaged schema changes + +A user may thus submit an `ALTER TABLE ... ALGORITHM=INSTANT` migration via `ApplySchema` with `direct` or `mysql` [strategies](../ddl-strategies), or plainly from their VTGate connection. + +## INSTANT DDL in managed schema changes + +With the `vitess` strategy, Vitess always prefers an Online DDL migration, which then allows [revertibility](../revertible-migrations). However, the `--prefer-instant-ddl` DDL strategy flag suggests to Vitess that, where possible, it should use `INSTANT` DDL. + +Vitess can predict whether a schema change is eligible to `INSTANT` DDL based on the existing table schema and the requested change. It is aware of the MySQL limitations (some of which are listed in the documentation) and can devise a plan for each distinct migration. If eligible for `INSTANT` DDL, and if `--prefer-instant-ddl` is specified, Vitess will automatically add `ALGORITHM=INSTANT` to the statement. Otherwise any `ALGORITHM` directive is ignored. + +If chosen to run as `INSTANT` DDL, the migration has no shadow table and is not revertible. + +## Example + +```sql + +mysql> set @@ddl_strategy='vitess --prefer-instant-ddl'; + +-- The migration is tracked, but the ALTER process won't start running. +mysql> alter table corder add column name varchar(32); ++--------------------------------------+ +| uuid | ++--------------------------------------+ +| f0070c6e_abe1_11ef_a809_b27afeff3c85 | ++--------------------------------------+ +``` + +Migrations executed with `--prefer-instant-ddl` are visible on `show vitess_migrations` just like any other migration: + + +```sql +mysql> show vitess_migrations like 'f0070c6e_abe1_11ef_a809_b27afeff3c85' \G +*************************** 1. row *************************** + id: 1 + migration_uuid: f0070c6e_abe1_11ef_a809_b27afeff3c85 + keyspace: commerce + shard: 0 + mysql_schema: vt_commerce + mysql_table: corder + migration_statement: alter table corder add column `name` varchar(32) + strategy: vitess + options: --prefer-instant-ddl + added_timestamp: 2024-11-26 12:33:55 + requested_timestamp: 2024-11-26 12:33:55 + ready_timestamp: NULL + started_timestamp: 2024-11-26 12:33:57 + liveness_timestamp: 2024-11-26 12:33:57 + completed_timestamp: 2024-11-26 12:33:56.652159 + cleanup_timestamp: NULL + migration_status: complete + log_path: + artifacts: + retries: 0 + tablet: zone1-0000000101 + tablet_failure: 0 + progress: 100 + migration_context: vtgate:e1131b44-abe1-11ef-a809-b27afeff3c85 + ddl_action: alter + message: +... + special_plan: {"operation":"instant-ddl"} +``` + +Note the migration has no `artifacts`, and that it is executed with `special_plan: {"operation":"instant-ddl"}`. + +## PARTITIONING notes + +MySQL partitioning changes sometimes exhibit behaviors similar to `INSTANT` DDL, although partition management really operates on entire hidden tables. + +Vitess specifically addresses the common use case of [`RANGE` partitioned](https://dev.mysql.com/doc/refman/en/partitioning-management-range-list.html) tables and partition rotation. + +The operation `ALTER TABLE ... ADD PARTITION` creates a new empty hidden table, which implements the new partition. The operation is as fast as `CREATE TABLE`. It is wasteful to run this operation with Online DDL because the existing data is completely unaffected. Vitess always opts to run `ADD PARTITION` directly against MySQL, much like an `INSTANT` DDL, and the operation is not revertible. This behavior is not controlled by any flags. + +The operation `ALTER TABLE ... DROP PARTITION` drops one or more partitions, hence effectively drops one or more tables imlementing those partitions. It would be a logical error to run this operation with Online DDL, because the intended outcome is to drop rows of data, where an Online DDL operation would just copy those rows of data onto the next compatible partition. Vitess thus always opts to run `DROP PARTITION` directly against MySQL, and the operation is not revertible. This behavior is likewise not controlled by any flags. + +Vitess does not offer any special behavior for other types of partitioning schemes and operations. \ No newline at end of file diff --git a/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md index d3a84ba1e..fe4ad1a83 100644 --- a/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: diff --git a/content/en/docs/20.0/reference/features/schema-management.md b/content/en/docs/20.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/20.0/reference/features/schema-management.md +++ b/content/en/docs/20.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md index 31abdc054..856cbce51 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md @@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order- - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md index aabf08cf1..4984aaddd 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md @@ -1,6 +1,6 @@ --- title: Concurrent migration execution -weight: 15 +weight: 16 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/'] --- diff --git a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md index 3dabe4915..2ebb9d95a 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md +++ b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md @@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic. - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table. +#### Notes and exceptions + +- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible. +- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes) + ### gh-ost [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool. diff --git a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md index 29207c877..69ca8d418 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -39,6 +39,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--retain-artifacts=`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value. - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. diff --git a/content/en/docs/20.0/user-guides/schema-changes/instant-ddl-migrations.md b/content/en/docs/20.0/user-guides/schema-changes/instant-ddl-migrations.md new file mode 100644 index 000000000..3ed7a37cd --- /dev/null +++ b/content/en/docs/20.0/user-guides/schema-changes/instant-ddl-migrations.md @@ -0,0 +1,88 @@ +--- +title: INSTANT DDL migrations +weight: 15 +aliases: ['/docs/user-guides/schema-changes/instant-ddl-migrations/'] +--- + +MySQL's [INSTANT DDL](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) is accessible in both unmanaged and managed vitess migrations. + +`INSTANT` DDL refers to `ALTER TABLE ... ALGORITHM=INSTANT` which runs near-instantaneously, though some locking is still witnessed in heavy workloads. It is limited to a subset of operations. The [documentation](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) lists the general limitations but does not cover the precise range of scenarios. + +A common naive approach to using `INSTANT` DDL is to optimistically attempt an `ALTER TABLE ... ALGORITHM=INSTANT`, and if that turns to be unsupported by `INSTANT` DDL, resort to standard `ALTER TABLE`. + +## INSTANT DDL in unmanaged schema changes + +A user may thus submit an `ALTER TABLE ... ALGORITHM=INSTANT` migration via `ApplySchema` with `direct` or `mysql` [strategies](../ddl-strategies), or plainly from their VTGate connection. + +## INSTANT DDL in managed schema changes + +With the `vitess` strategy, Vitess always prefers an Online DDL migration, which then allows [revertibility](../revertible-migrations). However, the `--prefer-instant-ddl` DDL strategy flag suggests to Vitess that, where possible, it should use `INSTANT` DDL. + +Vitess can predict whether a schema change is eligible to `INSTANT` DDL based on the existing table schema and the requested change. It is aware of the MySQL limitations (some of which are listed in the documentation) and can devise a plan for each distinct migration. If eligible for `INSTANT` DDL, and if `--prefer-instant-ddl` is specified, Vitess will automatically add `ALGORITHM=INSTANT` to the statement. Otherwise any `ALGORITHM` directive is ignored. + +If chosen to run as `INSTANT` DDL, the migration has no shadow table and is not revertible. + +## Example + +```sql + +mysql> set @@ddl_strategy='vitess --prefer-instant-ddl'; + +-- The migration is tracked, but the ALTER process won't start running. +mysql> alter table corder add column name varchar(32); ++--------------------------------------+ +| uuid | ++--------------------------------------+ +| f0070c6e_abe1_11ef_a809_b27afeff3c85 | ++--------------------------------------+ +``` + +Migrations executed with `--prefer-instant-ddl` are visible on `show vitess_migrations` just like any other migration: + + +```sql +mysql> show vitess_migrations like 'f0070c6e_abe1_11ef_a809_b27afeff3c85' \G +*************************** 1. row *************************** + id: 1 + migration_uuid: f0070c6e_abe1_11ef_a809_b27afeff3c85 + keyspace: commerce + shard: 0 + mysql_schema: vt_commerce + mysql_table: corder + migration_statement: alter table corder add column `name` varchar(32) + strategy: vitess + options: --prefer-instant-ddl + added_timestamp: 2024-11-26 12:33:55 + requested_timestamp: 2024-11-26 12:33:55 + ready_timestamp: NULL + started_timestamp: 2024-11-26 12:33:57 + liveness_timestamp: 2024-11-26 12:33:57 + completed_timestamp: 2024-11-26 12:33:56.652159 + cleanup_timestamp: NULL + migration_status: complete + log_path: + artifacts: + retries: 0 + tablet: zone1-0000000101 + tablet_failure: 0 + progress: 100 + migration_context: vtgate:e1131b44-abe1-11ef-a809-b27afeff3c85 + ddl_action: alter + message: +... + special_plan: {"operation":"instant-ddl"} +``` + +Note the migration has no `artifacts`, and that it is executed with `special_plan: {"operation":"instant-ddl"}`. + +## PARTITIONING notes + +MySQL partitioning changes sometimes exhibit behaviors similar to `INSTANT` DDL, although partition management really operates on entire hidden tables. + +Vitess specifically addresses the common use case of [`RANGE` partitioned](https://dev.mysql.com/doc/refman/en/partitioning-management-range-list.html) tables and partition rotation. + +The operation `ALTER TABLE ... ADD PARTITION` creates a new empty hidden table, which implements the new partition. The operation is as fast as `CREATE TABLE`. It is wasteful to run this operation with Online DDL because the existing data is completely unaffected. Vitess always opts to run `ADD PARTITION` directly against MySQL, much like an `INSTANT` DDL, and the operation is not revertible. This behavior is not controlled by any flags. + +The operation `ALTER TABLE ... DROP PARTITION` drops one or more partitions, hence effectively drops one or more tables imlementing those partitions. It would be a logical error to run this operation with Online DDL, because the intended outcome is to drop rows of data, where an Online DDL operation would just copy those rows of data onto the next compatible partition. Vitess thus always opts to run `DROP PARTITION` directly against MySQL, and the operation is not revertible. This behavior is likewise not controlled by any flags. + +Vitess does not offer any special behavior for other types of partitioning schemes and operations. \ No newline at end of file diff --git a/content/en/docs/20.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/20.0/user-guides/schema-changes/managed-online-schema-changes.md index 8497a6799..cda03ce16 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/20.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: diff --git a/content/en/docs/20.0/user-guides/schema-changes/validating-migrations.md b/content/en/docs/20.0/user-guides/schema-changes/validating-migrations.md index f6cec04bd..b3471a8f9 100644 --- a/content/en/docs/20.0/user-guides/schema-changes/validating-migrations.md +++ b/content/en/docs/20.0/user-guides/schema-changes/validating-migrations.md @@ -1,6 +1,6 @@ --- title: Validating schema migrations using `VDiff` -weight: 15 +weight: 17 aliases: [ '/docs/user-guides/schema-changes/validating-migrations/' ] --- diff --git a/content/en/docs/21.0/reference/features/schema-management.md b/content/en/docs/21.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/21.0/reference/features/schema-management.md +++ b/content/en/docs/21.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/21.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/21.0/user-guides/schema-changes/advanced-usage.md index 31abdc054..856cbce51 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/21.0/user-guides/schema-changes/advanced-usage.md @@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order- - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/21.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/21.0/user-guides/schema-changes/concurrent-migrations.md index aabf08cf1..4984aaddd 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/21.0/user-guides/schema-changes/concurrent-migrations.md @@ -1,6 +1,6 @@ --- title: Concurrent migration execution -weight: 15 +weight: 16 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/'] --- diff --git a/content/en/docs/21.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/21.0/user-guides/schema-changes/ddl-strategies.md index 3dabe4915..2ebb9d95a 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/ddl-strategies.md +++ b/content/en/docs/21.0/user-guides/schema-changes/ddl-strategies.md @@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic. - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table. +#### Notes and exceptions + +- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible. +- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes) + ### gh-ost [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool. diff --git a/content/en/docs/21.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/21.0/user-guides/schema-changes/ddl-strategy-flags.md index 29207c877..69ca8d418 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/21.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -39,6 +39,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--retain-artifacts=`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value. - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. diff --git a/content/en/docs/21.0/user-guides/schema-changes/instant-ddl-migrations.md b/content/en/docs/21.0/user-guides/schema-changes/instant-ddl-migrations.md new file mode 100644 index 000000000..3ed7a37cd --- /dev/null +++ b/content/en/docs/21.0/user-guides/schema-changes/instant-ddl-migrations.md @@ -0,0 +1,88 @@ +--- +title: INSTANT DDL migrations +weight: 15 +aliases: ['/docs/user-guides/schema-changes/instant-ddl-migrations/'] +--- + +MySQL's [INSTANT DDL](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) is accessible in both unmanaged and managed vitess migrations. + +`INSTANT` DDL refers to `ALTER TABLE ... ALGORITHM=INSTANT` which runs near-instantaneously, though some locking is still witnessed in heavy workloads. It is limited to a subset of operations. The [documentation](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) lists the general limitations but does not cover the precise range of scenarios. + +A common naive approach to using `INSTANT` DDL is to optimistically attempt an `ALTER TABLE ... ALGORITHM=INSTANT`, and if that turns to be unsupported by `INSTANT` DDL, resort to standard `ALTER TABLE`. + +## INSTANT DDL in unmanaged schema changes + +A user may thus submit an `ALTER TABLE ... ALGORITHM=INSTANT` migration via `ApplySchema` with `direct` or `mysql` [strategies](../ddl-strategies), or plainly from their VTGate connection. + +## INSTANT DDL in managed schema changes + +With the `vitess` strategy, Vitess always prefers an Online DDL migration, which then allows [revertibility](../revertible-migrations). However, the `--prefer-instant-ddl` DDL strategy flag suggests to Vitess that, where possible, it should use `INSTANT` DDL. + +Vitess can predict whether a schema change is eligible to `INSTANT` DDL based on the existing table schema and the requested change. It is aware of the MySQL limitations (some of which are listed in the documentation) and can devise a plan for each distinct migration. If eligible for `INSTANT` DDL, and if `--prefer-instant-ddl` is specified, Vitess will automatically add `ALGORITHM=INSTANT` to the statement. Otherwise any `ALGORITHM` directive is ignored. + +If chosen to run as `INSTANT` DDL, the migration has no shadow table and is not revertible. + +## Example + +```sql + +mysql> set @@ddl_strategy='vitess --prefer-instant-ddl'; + +-- The migration is tracked, but the ALTER process won't start running. +mysql> alter table corder add column name varchar(32); ++--------------------------------------+ +| uuid | ++--------------------------------------+ +| f0070c6e_abe1_11ef_a809_b27afeff3c85 | ++--------------------------------------+ +``` + +Migrations executed with `--prefer-instant-ddl` are visible on `show vitess_migrations` just like any other migration: + + +```sql +mysql> show vitess_migrations like 'f0070c6e_abe1_11ef_a809_b27afeff3c85' \G +*************************** 1. row *************************** + id: 1 + migration_uuid: f0070c6e_abe1_11ef_a809_b27afeff3c85 + keyspace: commerce + shard: 0 + mysql_schema: vt_commerce + mysql_table: corder + migration_statement: alter table corder add column `name` varchar(32) + strategy: vitess + options: --prefer-instant-ddl + added_timestamp: 2024-11-26 12:33:55 + requested_timestamp: 2024-11-26 12:33:55 + ready_timestamp: NULL + started_timestamp: 2024-11-26 12:33:57 + liveness_timestamp: 2024-11-26 12:33:57 + completed_timestamp: 2024-11-26 12:33:56.652159 + cleanup_timestamp: NULL + migration_status: complete + log_path: + artifacts: + retries: 0 + tablet: zone1-0000000101 + tablet_failure: 0 + progress: 100 + migration_context: vtgate:e1131b44-abe1-11ef-a809-b27afeff3c85 + ddl_action: alter + message: +... + special_plan: {"operation":"instant-ddl"} +``` + +Note the migration has no `artifacts`, and that it is executed with `special_plan: {"operation":"instant-ddl"}`. + +## PARTITIONING notes + +MySQL partitioning changes sometimes exhibit behaviors similar to `INSTANT` DDL, although partition management really operates on entire hidden tables. + +Vitess specifically addresses the common use case of [`RANGE` partitioned](https://dev.mysql.com/doc/refman/en/partitioning-management-range-list.html) tables and partition rotation. + +The operation `ALTER TABLE ... ADD PARTITION` creates a new empty hidden table, which implements the new partition. The operation is as fast as `CREATE TABLE`. It is wasteful to run this operation with Online DDL because the existing data is completely unaffected. Vitess always opts to run `ADD PARTITION` directly against MySQL, much like an `INSTANT` DDL, and the operation is not revertible. This behavior is not controlled by any flags. + +The operation `ALTER TABLE ... DROP PARTITION` drops one or more partitions, hence effectively drops one or more tables imlementing those partitions. It would be a logical error to run this operation with Online DDL, because the intended outcome is to drop rows of data, where an Online DDL operation would just copy those rows of data onto the next compatible partition. Vitess thus always opts to run `DROP PARTITION` directly against MySQL, and the operation is not revertible. This behavior is likewise not controlled by any flags. + +Vitess does not offer any special behavior for other types of partitioning schemes and operations. \ No newline at end of file diff --git a/content/en/docs/21.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/21.0/user-guides/schema-changes/managed-online-schema-changes.md index 8497a6799..cda03ce16 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/21.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: diff --git a/content/en/docs/21.0/user-guides/schema-changes/validating-migrations.md b/content/en/docs/21.0/user-guides/schema-changes/validating-migrations.md index f6cec04bd..b3471a8f9 100644 --- a/content/en/docs/21.0/user-guides/schema-changes/validating-migrations.md +++ b/content/en/docs/21.0/user-guides/schema-changes/validating-migrations.md @@ -1,6 +1,6 @@ --- title: Validating schema migrations using `VDiff` -weight: 15 +weight: 17 aliases: [ '/docs/user-guides/schema-changes/validating-migrations/' ] --- diff --git a/content/en/docs/22.0/reference/features/schema-management.md b/content/en/docs/22.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/22.0/reference/features/schema-management.md +++ b/content/en/docs/22.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/22.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/22.0/user-guides/schema-changes/advanced-usage.md index 31abdc054..856cbce51 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/22.0/user-guides/schema-changes/advanced-usage.md @@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order- - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/22.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/22.0/user-guides/schema-changes/concurrent-migrations.md index aabf08cf1..4984aaddd 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/22.0/user-guides/schema-changes/concurrent-migrations.md @@ -1,6 +1,6 @@ --- title: Concurrent migration execution -weight: 15 +weight: 16 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/'] --- diff --git a/content/en/docs/22.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/22.0/user-guides/schema-changes/ddl-strategies.md index 3dabe4915..2ebb9d95a 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/ddl-strategies.md +++ b/content/en/docs/22.0/user-guides/schema-changes/ddl-strategies.md @@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic. - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table. +#### Notes and exceptions + +- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible. +- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes) + ### gh-ost [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool. diff --git a/content/en/docs/22.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/22.0/user-guides/schema-changes/ddl-strategy-flags.md index 51219839c..37c18c431 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/22.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -40,6 +40,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--retain-artifacts=`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value. - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. diff --git a/content/en/docs/22.0/user-guides/schema-changes/instant-ddl-migrations.md b/content/en/docs/22.0/user-guides/schema-changes/instant-ddl-migrations.md new file mode 100644 index 000000000..3ed7a37cd --- /dev/null +++ b/content/en/docs/22.0/user-guides/schema-changes/instant-ddl-migrations.md @@ -0,0 +1,88 @@ +--- +title: INSTANT DDL migrations +weight: 15 +aliases: ['/docs/user-guides/schema-changes/instant-ddl-migrations/'] +--- + +MySQL's [INSTANT DDL](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) is accessible in both unmanaged and managed vitess migrations. + +`INSTANT` DDL refers to `ALTER TABLE ... ALGORITHM=INSTANT` which runs near-instantaneously, though some locking is still witnessed in heavy workloads. It is limited to a subset of operations. The [documentation](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) lists the general limitations but does not cover the precise range of scenarios. + +A common naive approach to using `INSTANT` DDL is to optimistically attempt an `ALTER TABLE ... ALGORITHM=INSTANT`, and if that turns to be unsupported by `INSTANT` DDL, resort to standard `ALTER TABLE`. + +## INSTANT DDL in unmanaged schema changes + +A user may thus submit an `ALTER TABLE ... ALGORITHM=INSTANT` migration via `ApplySchema` with `direct` or `mysql` [strategies](../ddl-strategies), or plainly from their VTGate connection. + +## INSTANT DDL in managed schema changes + +With the `vitess` strategy, Vitess always prefers an Online DDL migration, which then allows [revertibility](../revertible-migrations). However, the `--prefer-instant-ddl` DDL strategy flag suggests to Vitess that, where possible, it should use `INSTANT` DDL. + +Vitess can predict whether a schema change is eligible to `INSTANT` DDL based on the existing table schema and the requested change. It is aware of the MySQL limitations (some of which are listed in the documentation) and can devise a plan for each distinct migration. If eligible for `INSTANT` DDL, and if `--prefer-instant-ddl` is specified, Vitess will automatically add `ALGORITHM=INSTANT` to the statement. Otherwise any `ALGORITHM` directive is ignored. + +If chosen to run as `INSTANT` DDL, the migration has no shadow table and is not revertible. + +## Example + +```sql + +mysql> set @@ddl_strategy='vitess --prefer-instant-ddl'; + +-- The migration is tracked, but the ALTER process won't start running. +mysql> alter table corder add column name varchar(32); ++--------------------------------------+ +| uuid | ++--------------------------------------+ +| f0070c6e_abe1_11ef_a809_b27afeff3c85 | ++--------------------------------------+ +``` + +Migrations executed with `--prefer-instant-ddl` are visible on `show vitess_migrations` just like any other migration: + + +```sql +mysql> show vitess_migrations like 'f0070c6e_abe1_11ef_a809_b27afeff3c85' \G +*************************** 1. row *************************** + id: 1 + migration_uuid: f0070c6e_abe1_11ef_a809_b27afeff3c85 + keyspace: commerce + shard: 0 + mysql_schema: vt_commerce + mysql_table: corder + migration_statement: alter table corder add column `name` varchar(32) + strategy: vitess + options: --prefer-instant-ddl + added_timestamp: 2024-11-26 12:33:55 + requested_timestamp: 2024-11-26 12:33:55 + ready_timestamp: NULL + started_timestamp: 2024-11-26 12:33:57 + liveness_timestamp: 2024-11-26 12:33:57 + completed_timestamp: 2024-11-26 12:33:56.652159 + cleanup_timestamp: NULL + migration_status: complete + log_path: + artifacts: + retries: 0 + tablet: zone1-0000000101 + tablet_failure: 0 + progress: 100 + migration_context: vtgate:e1131b44-abe1-11ef-a809-b27afeff3c85 + ddl_action: alter + message: +... + special_plan: {"operation":"instant-ddl"} +``` + +Note the migration has no `artifacts`, and that it is executed with `special_plan: {"operation":"instant-ddl"}`. + +## PARTITIONING notes + +MySQL partitioning changes sometimes exhibit behaviors similar to `INSTANT` DDL, although partition management really operates on entire hidden tables. + +Vitess specifically addresses the common use case of [`RANGE` partitioned](https://dev.mysql.com/doc/refman/en/partitioning-management-range-list.html) tables and partition rotation. + +The operation `ALTER TABLE ... ADD PARTITION` creates a new empty hidden table, which implements the new partition. The operation is as fast as `CREATE TABLE`. It is wasteful to run this operation with Online DDL because the existing data is completely unaffected. Vitess always opts to run `ADD PARTITION` directly against MySQL, much like an `INSTANT` DDL, and the operation is not revertible. This behavior is not controlled by any flags. + +The operation `ALTER TABLE ... DROP PARTITION` drops one or more partitions, hence effectively drops one or more tables imlementing those partitions. It would be a logical error to run this operation with Online DDL, because the intended outcome is to drop rows of data, where an Online DDL operation would just copy those rows of data onto the next compatible partition. Vitess thus always opts to run `DROP PARTITION` directly against MySQL, and the operation is not revertible. This behavior is likewise not controlled by any flags. + +Vitess does not offer any special behavior for other types of partitioning schemes and operations. \ No newline at end of file diff --git a/content/en/docs/22.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/22.0/user-guides/schema-changes/managed-online-schema-changes.md index 8497a6799..cda03ce16 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/22.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: diff --git a/content/en/docs/22.0/user-guides/schema-changes/table-lifecycle.md b/content/en/docs/22.0/user-guides/schema-changes/table-lifecycle.md index 0da438713..f1f01a64d 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/table-lifecycle.md +++ b/content/en/docs/22.0/user-guides/schema-changes/table-lifecycle.md @@ -88,7 +88,7 @@ Starting in Vitess `v20`, the table naming format will change. Tables will be na - `_vt_evc_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_` - `_vt_drp_6ace8bcef73211ea87e9f875a4d24e90_20200915120410_` -`v19` supports the new naming format, but does not generate any tables in this format. `v20` will generate tables in the new format, and will support the old format. Support for old format will be dropped in `v21` or later. +`v19` supports the new naming format, but does not generate any tables in this format. `v20` will generate tables in the new format, and will support the old format. Support for old format will be dropped in `v22` or later. {{< /info >}} ## Automated lifecycle diff --git a/content/en/docs/22.0/user-guides/schema-changes/validating-migrations.md b/content/en/docs/22.0/user-guides/schema-changes/validating-migrations.md index f6cec04bd..b3471a8f9 100644 --- a/content/en/docs/22.0/user-guides/schema-changes/validating-migrations.md +++ b/content/en/docs/22.0/user-guides/schema-changes/validating-migrations.md @@ -1,6 +1,6 @@ --- title: Validating schema migrations using `VDiff` -weight: 15 +weight: 17 aliases: [ '/docs/user-guides/schema-changes/validating-migrations/' ] --- diff --git a/content/en/docs/archive/17.0/reference/features/schema-management.md b/content/en/docs/archive/17.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/archive/17.0/reference/features/schema-management.md +++ b/content/en/docs/archive/17.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/archive/17.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/archive/17.0/user-guides/schema-changes/advanced-usage.md index 434193192..7820b10a3 100644 --- a/content/en/docs/archive/17.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/archive/17.0/user-guides/schema-changes/advanced-usage.md @@ -178,3 +178,14 @@ $ vtctldclient ApplySchema --skip_preflight --ddl_strategy='vitess --allow-concu - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/archive/17.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/archive/17.0/user-guides/schema-changes/ddl-strategy-flags.md index 8e13cef05..288ad00c3 100644 --- a/content/en/docs/archive/17.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/archive/17.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -31,6 +31,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. - `--singleton-context`: only allow migrations submitted under same _context_ to be pending at any given time. Migrations submitted with a different _context_ are rejected for as long as at least one of the initially submitted migrations is pending. diff --git a/content/en/docs/archive/17.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/archive/17.0/user-guides/schema-changes/managed-online-schema-changes.md index e5efebfff..66fd3824d 100644 --- a/content/en/docs/archive/17.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/archive/17.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: diff --git a/content/en/docs/archive/18.0/reference/features/schema-management.md b/content/en/docs/archive/18.0/reference/features/schema-management.md index 5a12afb27..af3100d52 100644 --- a/content/en/docs/archive/18.0/reference/features/schema-management.md +++ b/content/en/docs/archive/18.0/reference/features/schema-management.md @@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/) * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/) * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/) +* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/) The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql= || --sql_file=} ` diff --git a/content/en/docs/archive/18.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/archive/18.0/user-guides/schema-changes/advanced-usage.md index 31abdc054..856cbce51 100644 --- a/content/en/docs/archive/18.0/user-guides/schema-changes/advanced-usage.md +++ b/content/en/docs/archive/18.0/user-guides/schema-changes/advanced-usage.md @@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order- - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion. - in-order completion also works with `--postpone-launch` and `--postpone-completion`. {{< /info >}} + +## INSTANT DDL + +Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example: + + +```sh +$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce +``` + +You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration. diff --git a/content/en/docs/archive/18.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/archive/18.0/user-guides/schema-changes/concurrent-migrations.md index aabf08cf1..4984aaddd 100644 --- a/content/en/docs/archive/18.0/user-guides/schema-changes/concurrent-migrations.md +++ b/content/en/docs/archive/18.0/user-guides/schema-changes/concurrent-migrations.md @@ -1,6 +1,6 @@ --- title: Concurrent migration execution -weight: 15 +weight: 16 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/'] --- diff --git a/content/en/docs/archive/18.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/archive/18.0/user-guides/schema-changes/ddl-strategy-flags.md index a950b42c5..81c15f332 100644 --- a/content/en/docs/archive/18.0/user-guides/schema-changes/ddl-strategy-flags.md +++ b/content/en/docs/archive/18.0/user-guides/schema-changes/ddl-strategy-flags.md @@ -33,6 +33,8 @@ Vitess respects the following flags. They can be combined unless specifically in - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations). +- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible. + - `--retain-artifacts=`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value. - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked. diff --git a/content/en/docs/archive/18.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/archive/18.0/user-guides/schema-changes/managed-online-schema-changes.md index d3a84ba1e..fe4ad1a83 100644 --- a/content/en/docs/archive/18.0/user-guides/schema-changes/managed-online-schema-changes.md +++ b/content/en/docs/archive/18.0/user-guides/schema-changes/managed-online-schema-changes.md @@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently - Support for [postponed migrations](../postponed-migrations/) - Support for [failover agnostic migrations](../recoverable-migrations/) - Support for [concurrent migrations](../concurrent-migrations/) +- Support for [INSTANT DDL](../instant-ddl-migrations/) - See also [advanced usage](../advanced-usage/) As general overview: