Docs: INSTANT DDL and RANGE PARTITIONING support for managed sche…

…ma migrations (#1899) * document '--prefer-instant-ddl' DDL strategy flag Signed-off-by: Shlomi Noach <[email protected]> * clarify removal of old table format Signed-off-by: Shlomi Noach <[email protected]> * fix ordering of schema change pages Signed-off-by: Shlomi Noach <[email protected]> * preemptively add link to instant DDL page Signed-off-by: Shlomi Noach <[email protected]> * preemptively add more link to instant DDL page Signed-off-by: Shlomi Noach <[email protected]> * Adding INSTANT DDL docs for v22 Signed-off-by: Shlomi Noach <[email protected]> * INSTANT DDL in advanced usage Signed-off-by: Shlomi Noach <[email protected]> * formatting Signed-off-by: Shlomi Noach <[email protected]> * add notes and exceptions Signed-off-by: Shlomi Noach <[email protected]> * partitioning notes Signed-off-by: Shlomi Noach <[email protected]> * instant DDL docs for older versions Signed-off-by: Shlomi Noach <[email protected]> * typo Signed-off-by: Shlomi Noach <[email protected]> * fix excessive text Signed-off-by: Shlomi Noach <[email protected]> * remove /8.0/ from urls Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]>
vitessio · Dec 3, 2024 · c6e15b2 · c6e15b2
1 parent 0108e72
commit c6e15b2
Show file tree

Hide file tree

Showing 41 changed files with 471 additions and 9 deletions.
diff --git a/content/en/docs/19.0/reference/features/schema-management.md b/content/en/docs/19.0/reference/features/schema-management.md
@@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man
 * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/)
 * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/)
 * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/)
+* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/)
 
 The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql=<sql> || --sql_file=<filename>} <keyspace>`
 

diff --git a/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/19.0/user-guides/schema-changes/advanced-usage.md
@@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order-
 - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion.
 - in-order completion also works with `--postpone-launch` and `--postpone-completion`.
 {{< /info >}}
+
+## INSTANT DDL
+
+Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example:
+
+
+```sh
+$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce
+```
+
+You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration.
diff --git a/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/19.0/user-guides/schema-changes/concurrent-migrations.md
@@ -1,6 +1,6 @@
 ---
 title: Concurrent migration execution
-weight: 15
+weight: 16
 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/']
 ---
 

diff --git a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategies.md
@@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc
 - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic.
 - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table.
 
+#### Notes and exceptions
+
+- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible.
+- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes)
+
 ### gh-ost
 
 [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool.

diff --git a/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/19.0/user-guides/schema-changes/ddl-strategy-flags.md
@@ -39,6 +39,8 @@ Vitess respects the following flags. They can be combined unless specifically in
 
 - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations).
 
+- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible.
+
 - `--retain-artifacts=<duration>`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value.
 
 - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked.

diff --git a/content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md b/content/en/docs/19.0/user-guides/schema-changes/instant-ddl-migrations.md
@@ -0,0 +1,88 @@
+---
+title: INSTANT DDL migrations
+weight: 15
+aliases: ['/docs/user-guides/schema-changes/instant-ddl-migrations/']
+---
+
+MySQL's [INSTANT DDL](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) is accessible in both unmanaged and managed vitess migrations.
+
+`INSTANT` DDL refers to `ALTER TABLE ... ALGORITHM=INSTANT` which runs near-instantaneously, though some locking is still witnessed in heavy workloads. It is limited to a subset of operations. The [documentation](https://dev.mysql.com/doc/refman/en/innodb-online-ddl-operations.html) lists the general limitations but does not cover the precise range of scenarios.
+
+A common naive approach to using `INSTANT` DDL is to optimistically attempt an `ALTER TABLE ... ALGORITHM=INSTANT`, and if that turns to be unsupported by `INSTANT` DDL, resort to standard `ALTER TABLE`.
+
+## INSTANT DDL in unmanaged schema changes
+
+A user may thus submit an `ALTER TABLE ... ALGORITHM=INSTANT` migration via `ApplySchema` with `direct` or `mysql` [strategies](../ddl-strategies), or plainly from their VTGate connection.
+
+## INSTANT DDL in managed schema changes
+
+With the `vitess` strategy, Vitess always prefers an Online DDL migration, which then allows [revertibility](../revertible-migrations). However, the `--prefer-instant-ddl` DDL strategy flag suggests to Vitess that, where possible, it should use `INSTANT` DDL.
+
+Vitess can predict whether a schema change is eligible to `INSTANT` DDL based on the existing table schema and the requested change. It is aware of the MySQL limitations (some of which are listed in the documentation) and can devise a plan for each distinct migration. If eligible for `INSTANT` DDL, and if `--prefer-instant-ddl` is specified, Vitess will automatically add `ALGORITHM=INSTANT` to the statement. Otherwise any `ALGORITHM` directive is ignored.
+
+If chosen to run as `INSTANT` DDL, the migration has no shadow table and is not revertible.
+
+## Example
+
+```sql
+
+mysql> set @@ddl_strategy='vitess --prefer-instant-ddl';
+
+-- The migration is tracked, but the ALTER process won't start running.
+mysql> alter table corder add column name varchar(32);
++--------------------------------------+
+| uuid                                 |
++--------------------------------------+
+| f0070c6e_abe1_11ef_a809_b27afeff3c85 |
++--------------------------------------+
+```
+
+Migrations executed with `--prefer-instant-ddl` are visible on `show vitess_migrations` just like any other migration:
+
+
+```sql
+mysql> show vitess_migrations like 'f0070c6e_abe1_11ef_a809_b27afeff3c85' \G
+*************************** 1. row ***************************
+                             id: 1
+                 migration_uuid: f0070c6e_abe1_11ef_a809_b27afeff3c85
+                       keyspace: commerce
+                          shard: 0
+                   mysql_schema: vt_commerce
+                    mysql_table: corder
+            migration_statement: alter table corder add column `name` varchar(32)
+                       strategy: vitess
+                        options: --prefer-instant-ddl
+                added_timestamp: 2024-11-26 12:33:55
+            requested_timestamp: 2024-11-26 12:33:55
+                ready_timestamp: NULL
+              started_timestamp: 2024-11-26 12:33:57
+             liveness_timestamp: 2024-11-26 12:33:57
+            completed_timestamp: 2024-11-26 12:33:56.652159
+              cleanup_timestamp: NULL
+               migration_status: complete
+                       log_path:
+                      artifacts:
+                        retries: 0
+                         tablet: zone1-0000000101
+                 tablet_failure: 0
+                       progress: 100
+              migration_context: vtgate:e1131b44-abe1-11ef-a809-b27afeff3c85
+                     ddl_action: alter
+                        message:
+...
+                   special_plan: {"operation":"instant-ddl"}
+```
+
+Note the migration has no `artifacts`, and that it is executed with `special_plan: {"operation":"instant-ddl"}`.
+
+## PARTITIONING notes
+
+MySQL partitioning changes sometimes exhibit behaviors similar to `INSTANT` DDL, although partition management really operates on entire hidden tables.
+
+Vitess specifically addresses the common use case of [`RANGE` partitioned](https://dev.mysql.com/doc/refman/en/partitioning-management-range-list.html) tables and partition rotation.
+
+The operation `ALTER TABLE ... ADD PARTITION` creates a new empty hidden table, which implements the new partition. The operation is as fast as `CREATE TABLE`. It is wasteful to run this operation with Online DDL because the existing data is completely unaffected. Vitess always opts to run `ADD PARTITION` directly against MySQL, much like an `INSTANT` DDL, and the operation is not revertible. This behavior is not controlled by any flags.
+
+The operation `ALTER TABLE ... DROP PARTITION` drops one or more partitions, hence effectively drops one or more tables imlementing those partitions. It would be a logical error to run this operation with Online DDL, because the intended outcome is to drop rows of data, where an Online DDL operation would just copy those rows of data onto the next compatible partition. Vitess thus always opts to run `DROP PARTITION` directly against MySQL, and the operation is not revertible. This behavior is likewise not controlled by any flags.
+
+Vitess does not offer any special behavior for other types of partitioning schemes and operations.
diff --git a/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md b/content/en/docs/19.0/user-guides/schema-changes/managed-online-schema-changes.md
@@ -16,6 +16,7 @@ Vitess offers managed, online schema migrations (aka Online DDL), transparently
 - Support for [postponed migrations](../postponed-migrations/)
 - Support for [failover agnostic migrations](../recoverable-migrations/)
 - Support for [concurrent migrations](../concurrent-migrations/)
+- Support for [INSTANT DDL](../instant-ddl-migrations/)
 - See also [advanced usage](../advanced-usage/)
 
 As general overview:

diff --git a/content/en/docs/20.0/reference/features/schema-management.md b/content/en/docs/20.0/reference/features/schema-management.md
@@ -90,6 +90,7 @@ Vitess offers [managed schema migration](../../../user-guides/schema-changes/man
 * Support for [postponed migrations](../../../user-guides/schema-changes/postponed-migrations/)
 * Support for [failover agnostic migrations](../../../user-guides/schema-changes/recoverable-migrations/)
 * Support for [concurrent migrations](../../../user-guides/schema-changes/concurrent-migrations/)
+* Support for [INSTANT DDL](../../../user-guides/schema-changes/instant-ddl-migrations/)
 
 The [ApplySchema](../../../reference/programs/vtctl/schema-version-permissions/#applyschema) command applies a schema change to the specified keyspace on all shards. The command format is: `ApplySchema -- {--sql=<sql> || --sql_file=<filename>} <keyspace>`
 

diff --git a/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md b/content/en/docs/20.0/user-guides/schema-changes/advanced-usage.md
@@ -188,3 +188,14 @@ $ vtctldclient ApplySchema --ddl-strategy='vitess --allow-concurrent --in-order-
 - `--allow-concurrent` is optional, but is likely to be the main use case for using in-order completion.
 - in-order completion also works with `--postpone-launch` and `--postpone-completion`.
 {{< /info >}}
+
+## INSTANT DDL
+
+Vitess can predict ahead of time whether a migration is eligible for `ALGORITHM=INSTANT`. As such, you may submit a migration opting in to `INSTANT` DDL if applicable. In such case, Vitess adds `ALGORITHM=INSTANT` to your migration. Otherwise, it runs an Online DDL migration. Example:
+
+
+```sh
+$ vtctldclient ApplySchema --ddl-strategy='vitess --prefer-instant-ddl' --sql "alter table corder add column name varchar(32); alter table users add unique key login_idx (login);" commerce
+```
+
+You may submit multiple migrations with the `--prefer-instant-ddl` DDL strategy flag. Vitess will determine per migration whether `INSTANT` DDL is applicable or not. In the above example, it will apply `INSTANT` DDL onto the `corder` migration, but will run a `vitess` Online DDL for the `users` migration.
diff --git a/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md b/content/en/docs/20.0/user-guides/schema-changes/concurrent-migrations.md
@@ -1,6 +1,6 @@
 ---
 title: Concurrent migration execution
-weight: 15
+weight: 16
 aliases: ['/docs/user-guides/schema-changes/concurrent-migrations/']
 ---
 

diff --git a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategies.md
@@ -79,6 +79,11 @@ The `vitess` strategy invokes Vitess's built in [VReplication](../../../referenc
 - Support cut-over backoff: should a cut-over fail due to timeout, next cut-overs take place at increasing intervals and up to `30min` intevals, so as to not overwhelm production traffic.
 - Support forced cut-over, to prioritise completion of the migration over any queries using the mgirated table, or over any transactions holding locks on the table.
 
+#### Notes and exceptions
+
+- `vitess` migrations support `INSTANT` DDL where applicable. See [INSTANT DDL](../instant-ddl-migrations/). A migration that runs with `ALGORITHM=INSTANT` does not use a shadow table and is not revertible.
+- `RANGE` partitioning rotation: `ADD PARTITION` and `DROP PARTITION` statements are executed directly against MySQL and not as a `vreplication` Online DDL. See [PARTITIONING notes](../instant-ddl-migrations/#partitioning-notes)
+
 ### gh-ost
 
 [gh-ost](https://github.com/github/gh-ost) was developed by [GitHub](https://github.com) as a lightweight and safe schema migration tool.

diff --git a/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md b/content/en/docs/20.0/user-guides/schema-changes/ddl-strategy-flags.md
@@ -39,6 +39,8 @@ Vitess respects the following flags. They can be combined unless specifically in
 
 - `--postpone-launch`: initiate a migration that remains `queued` and only launches per user command. See [postponed migrations](../postponed-migrations).
 
+- `--prefer-instant-ddl`: where possible, apply `ALGORITHM=INSTANT` to the migration. This is applicable to `ALTER TABLE` migrations with `vitess` strategy. Vitess pre-computes whether the migration is eligible for `INSTANT` DDL. The [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html) references an incomplete list of eligible changes. If applicable, vitess does not create a shadow table and the migration is not revertible.
+
 - `--retain-artifacts=<duration>`: set an explicit artifact retention for this migration. If nonzero, this value overrides the `vttablet --retain_online_ddl_tables` value.
 
 - `--singleton`: only allow a single pending migration to be submitted at a time. From the moment the migration is queued, and until either completion, failure or cancellation, no other new `--singleton` migration can be submitted. New requests will be rejected with error. `--singleton` works as a an exclusive lock for pending migrations. Note that this only affects migrations with `--singleton` flag. Migrations running without that flag are unaffected and unblocked.