diff --git a/changelog/15.0/15.0.5/changelog.md b/changelog/15.0/15.0.5/changelog.md new file mode 100644 index 00000000000..07ac0749677 --- /dev/null +++ b/changelog/15.0/15.0.5/changelog.md @@ -0,0 +1,40 @@ +# Changelog of Vitess v15.0.5 + +### Bug fixes +#### Build/CI + * [release-15.0] Make `Static Code Checks Etc` fail if the `./changelog` folder is out-of-date (#14003) [#14004](https://github.com/vitessio/vitess/pull/14004) + * [release-15.0] Enable failures in `tools/e2e_test_race.sh` and fix races (#13654) [#14009](https://github.com/vitessio/vitess/pull/14009) +#### Cluster management + * [release-15.0] Fix `BackupShard` to get its options from its own flags (#13813) [#13818](https://github.com/vitessio/vitess/pull/13818) +#### Online DDL + * v15 backport: Solve RevertMigration.Comment read/write concurrency issue [#13735](https://github.com/vitessio/vitess/pull/13735) + * v15 backport: Onlineddl: formalize "immediate operations", respect --postpone-completion strategy flag [#13832](https://github.com/vitessio/vitess/pull/13832) + * [release-15.0] OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations (#13928) [#13935](https://github.com/vitessio/vitess/pull/13935) + * [release-15.0] OnlineDDL: cleanup cancelled migration artifacts; support `--retain-artifacts=` DDL strategy flag (#14029) [#14035](https://github.com/vitessio/vitess/pull/14035) + * [release-15.0] bugfix: change column name and type to json (#14093) [#14115](https://github.com/vitessio/vitess/pull/14115) +#### Query Serving + * [release-15.0] vtgate: fix race condition iterating tables and views from schema tracker (#13673) [#13794](https://github.com/vitessio/vitess/pull/13794) + * [15.0] bugfixes: collection of fixes to bugs found while fuzzing [#13807](https://github.com/vitessio/vitess/pull/13807) + * [release-15.0] fix data race in join engine primitive olap streaming mode execution (#14012) [#14014](https://github.com/vitessio/vitess/pull/14014) + * [release-15.0] fix: cost to include subshard opcode (#14023) [#14025](https://github.com/vitessio/vitess/pull/14025) +#### TabletManager + * [release-15.0] Fix: convertBoolToSemiSyncAction method to account for all semi sync actions (#13075) [#13957](https://github.com/vitessio/vitess/pull/13957) +### CI/Build +#### Documentation + * [release-15.0] update docgen to embed commit ID in autogenerated doc frontmatter (#14056) [#14072](https://github.com/vitessio/vitess/pull/14072) +#### VTorc + * [release-15.0] docker: add dedicated vtorc container (#14126) [#14145](https://github.com/vitessio/vitess/pull/14145) +### Documentation +#### Documentation + * [release-15.0] anonymize homedirs in generated docs (#14101) [#14104](https://github.com/vitessio/vitess/pull/14104) +### Internal Cleanup +#### Build/CI + * [release-15.0] Bump upgrade test to `go1.20.8` [#13938](https://github.com/vitessio/vitess/pull/13938) +### Release +#### General + * Code freeze of release-15.0 for `v15.0.5` [#14141](https://github.com/vitessio/vitess/pull/14141) +### Testing +#### Build/CI + * [release-15.0] Flakes: Delete VTDATAROOT files in reparent test teardown within CI (#13793) [#14118](https://github.com/vitessio/vitess/pull/14118) + * [release-15.0] Remove FOSSA Test from CI until we can do it in a secure way (#14119) [#14120](https://github.com/vitessio/vitess/pull/14120) + diff --git a/changelog/15.0/15.0.5/release_notes.md b/changelog/15.0/15.0.5/release_notes.md new file mode 100644 index 00000000000..34f9da437bd --- /dev/null +++ b/changelog/15.0/15.0.5/release_notes.md @@ -0,0 +1,7 @@ +# Release of Vitess v15.0.5 +The entire changelog for this release can be found [here](https://github.com/vitessio/vitess/blob/main/changelog/15.0/15.0.5/changelog.md). + +The release includes 20 merged Pull Requests. + +Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @arthurschreiber, @frouioui, @shlomi-noach, @systay + diff --git a/changelog/15.0/README.md b/changelog/15.0/README.md index 17807db2ebc..cc4a6caa0f0 100644 --- a/changelog/15.0/README.md +++ b/changelog/15.0/README.md @@ -1,5 +1,9 @@ ## v15.0 The dedicated team for this release can be found [here](team.md). +* **[15.0.5](15.0.5)** + * [Changelog](15.0.5/changelog.md) + * [Release Notes](15.0.5/release_notes.md) + * **[15.0.4](15.0.4)** * [Changelog](15.0.4/changelog.md) * [Release Notes](15.0.4/release_notes.md) diff --git a/changelog/16.0/16.0.5/changelog.md b/changelog/16.0/16.0.5/changelog.md new file mode 100644 index 00000000000..d5a13b56c02 --- /dev/null +++ b/changelog/16.0/16.0.5/changelog.md @@ -0,0 +1,55 @@ +# Changelog of Vitess v16.0.5 + +### Bug fixes +#### Build/CI + * [release-16.0] Make `Static Code Checks Etc` fail if the `./changelog` folder is out-of-date (#14003) [#14005](https://github.com/vitessio/vitess/pull/14005) + * [release-16.0] Enable failures in `tools/e2e_test_race.sh` and fix races (#13654) [#14010](https://github.com/vitessio/vitess/pull/14010) +#### Cluster management + * [release-16.0] Fix `BackupShard` to get its options from its own flags (#13813) [#13819](https://github.com/vitessio/vitess/pull/13819) + * [release-16.0] Fix: convertBoolToSemiSyncAction method to account for all semi sync actions (#13075) [#13958](https://github.com/vitessio/vitess/pull/13958) +#### Online DDL + * [release-16.0] OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations (#13928) [#13936](https://github.com/vitessio/vitess/pull/13936) + * [release-16.0] OnlineDDL: cleanup cancelled migration artifacts; support `--retain-artifacts=` DDL strategy flag (#14029) [#14036](https://github.com/vitessio/vitess/pull/14036) + * [release-16.0] TableGC: support DROP VIEW (#14020) [#14044](https://github.com/vitessio/vitess/pull/14044) + * [release-16.0] bugfix: change column name and type to json (#14093) [#14116](https://github.com/vitessio/vitess/pull/14116) +#### Query Serving + * [release-16.0] Fix for "text type with an unknown/unsupported collation cannot be hashed" error (#13852) [#13863](https://github.com/vitessio/vitess/pull/13863) + * [release-16.0] handle large number of predicates without timing out (#13979) [#13981](https://github.com/vitessio/vitess/pull/13981) + * [release-16.0] fix data race in join engine primitive olap streaming mode execution (#14012) [#14015](https://github.com/vitessio/vitess/pull/14015) + * [release-16.0] fix: cost to include subshard opcode (#14023) [#14026](https://github.com/vitessio/vitess/pull/14026) +#### VReplication + * [release-16.0] Flakes: skip flaky check that ETA for a VReplication VDiff2 Progress command is in the future. (#13804) [#13816](https://github.com/vitessio/vitess/pull/13816) + * [release-16.0] copy over existing vreplication rows copied to local counter if resuming from another tablet (#13949) [#13962](https://github.com/vitessio/vitess/pull/13962) + * [release-16.0] VDiff: correct handling of default source and target cells (#13969) [#13983](https://github.com/vitessio/vitess/pull/13983) + * Backport [release-16.0] Flakes: Add recently added 'select rows_copied' query to ignore list #13993 [#14040](https://github.com/vitessio/vitess/pull/14040) + * [release-16.0] VDiff: properly split cell values in record when using TabletPicker (#14099) [#14102](https://github.com/vitessio/vitess/pull/14102) + * [release-16.0] VDiff: Cleanup the controller for a VDiff before deleting it (#14107) [#14124](https://github.com/vitessio/vitess/pull/14124) +#### VTCombo + * [release-16.0] Fix vtcombo DBDDL plugin race condition (#13117) [#13741](https://github.com/vitessio/vitess/pull/13741) +### CI/Build +#### Documentation + * [release-16.0] update docgen to embed commit ID in autogenerated doc frontmatter (#14056) [#14073](https://github.com/vitessio/vitess/pull/14073) +#### General + * [release-16.0] Upgrade the Golang version to `go1.20.8` [#13932](https://github.com/vitessio/vitess/pull/13932) +#### VTorc + * [release-16.0] docker: add dedicated vtorc container (#14126) [#14146](https://github.com/vitessio/vitess/pull/14146) +### Documentation +#### Documentation + * [release-16.0] anonymize homedirs in generated docs (#14101) [#14105](https://github.com/vitessio/vitess/pull/14105) +### Enhancement +#### VReplication + * [release-16.0] MoveTables: add flag to specify that routing rules should not be created when a movetables workflow is created [#13858](https://github.com/vitessio/vitess/pull/13858) +### Internal Cleanup +#### Build/CI + * [release-16.0] Use Debian Bullseye in Bootstrap (#13757) [#13921](https://github.com/vitessio/vitess/pull/13921) +#### Query Serving + * [release-16.0] moved timeout test to different package (#14028) [#14031](https://github.com/vitessio/vitess/pull/14031) +### Release +#### General + * Code freeze of `release-16.0` for `v16.0.5` [#14132](https://github.com/vitessio/vitess/pull/14132) +### Testing +#### Build/CI + * [release-16.0] Remove FOSSA Test from CI until we can do it in a secure way (#14119) [#14121](https://github.com/vitessio/vitess/pull/14121) +#### VReplication + * [release-16.0] Flakes: empty vtdataroot before starting a new vreplication e2e test (#13803) [#13821](https://github.com/vitessio/vitess/pull/13821) + diff --git a/changelog/16.0/16.0.5/release_notes.md b/changelog/16.0/16.0.5/release_notes.md new file mode 100644 index 00000000000..e988f5fdbd0 --- /dev/null +++ b/changelog/16.0/16.0.5/release_notes.md @@ -0,0 +1,7 @@ +# Release of Vitess v16.0.5 +The entire changelog for this release can be found [here](https://github.com/vitessio/vitess/blob/main/changelog/16.0/16.0.5/changelog.md). + +The release includes 29 merged Pull Requests. + +Thanks to all our contributors: @GuptaManan100, @app/github-actions, @app/vitess-bot, @arthurschreiber, @rohit-nayak-ps + diff --git a/changelog/16.0/16.0.5/summary.md b/changelog/16.0/16.0.5/summary.md new file mode 100644 index 00000000000..e69de29bb2d diff --git a/changelog/16.0/README.md b/changelog/16.0/README.md index 75b3f3a0a1f..d45a817ad48 100644 --- a/changelog/16.0/README.md +++ b/changelog/16.0/README.md @@ -1,5 +1,9 @@ ## v16.0 The dedicated team for this release can be found [here](team.md). +* **[16.0.5](16.0.5)** + * [Changelog](16.0.5/changelog.md) + * [Release Notes](16.0.5/release_notes.md) + * **[16.0.4](16.0.4)** * [Changelog](16.0.4/changelog.md) * [Release Notes](16.0.4/release_notes.md) diff --git a/changelog/17.0/17.0.3/changelog.md b/changelog/17.0/17.0.3/changelog.md new file mode 100644 index 00000000000..f3f5f3e261f --- /dev/null +++ b/changelog/17.0/17.0.3/changelog.md @@ -0,0 +1,62 @@ +# Changelog of Vitess v17.0.3 + +### Bug fixes +#### Build/CI + * [release-17.0] Make `Static Code Checks Etc` fail if the `./changelog` folder is out-of-date (#14003) [#14006](https://github.com/vitessio/vitess/pull/14006) + * [release-17.0] Enable failures in `tools/e2e_test_race.sh` and fix races (#13654) [#14011](https://github.com/vitessio/vitess/pull/14011) +#### CLI + * [release-17.0] backport mysqlctl CLI compatibility fix to 17.0 [#14082](https://github.com/vitessio/vitess/pull/14082) +#### Cluster management + * [release-17.0] Fix `BackupShard` to get its options from its own flags (#13813) [#13820](https://github.com/vitessio/vitess/pull/13820) +#### Evalengine + * [release-17.0] evalengine: Mark UUID() function as non-constant (#14051) [#14057](https://github.com/vitessio/vitess/pull/14057) +#### Online DDL + * [release-17.0] OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations (#13928) [#13937](https://github.com/vitessio/vitess/pull/13937) + * [release-17.0] OnlineDDL: cleanup cancelled migration artifacts; support `--retain-artifacts=` DDL strategy flag (#14029) [#14037](https://github.com/vitessio/vitess/pull/14037) + * [release-17.0] TableGC: support DROP VIEW (#14020) [#14045](https://github.com/vitessio/vitess/pull/14045) + * [release-17.0] bugfix: change column name and type to json (#14093) [#14117](https://github.com/vitessio/vitess/pull/14117) +#### Query Serving + * [release-17.0] Rewrite `USING` to `ON` condition for joins (#13931) [#13941](https://github.com/vitessio/vitess/pull/13941) + * [release-17.0] handle large number of predicates without timing out (#13979) [#13982](https://github.com/vitessio/vitess/pull/13982) + * [release-17.0] fix data race in join engine primitive olap streaming mode execution (#14012) [#14016](https://github.com/vitessio/vitess/pull/14016) + * [release-17.0] fix: cost to include subshard opcode (#14023) [#14027](https://github.com/vitessio/vitess/pull/14027) + * [release-17.0] Add session flag for stream execute grpc api (#14046) [#14053](https://github.com/vitessio/vitess/pull/14053) +#### Throttler + * [release-17.0] Tablet throttler: empty list of probes on non-leader (#13926) [#13952](https://github.com/vitessio/vitess/pull/13952) +#### VReplication + * [release-17.0] Flakes: skip flaky check that ETA for a VReplication VDiff2 Progress command is in the future. (#13804) [#13817](https://github.com/vitessio/vitess/pull/13817) + * [release-17.0] VReplication: Handle SQL NULL and JSON 'null' correctly for JSON columns (#13944) [#13947](https://github.com/vitessio/vitess/pull/13947) + * [release-17.0] copy over existing vreplication rows copied to local counter if resuming from another tablet (#13949) [#13963](https://github.com/vitessio/vitess/pull/13963) + * [release-17.0] VDiff: correct handling of default source and target cells (#13969) [#13984](https://github.com/vitessio/vitess/pull/13984) + * Backport [release-17.0] Flakes: Add recently added 'select rows_copied' query to ignore list #13993 [#14039](https://github.com/vitessio/vitess/pull/14039) + * [release-17.0] json: Fix quoting JSON keys (#14066) [#14068](https://github.com/vitessio/vitess/pull/14068) + * [release-17.0] VDiff: properly split cell values in record when using TabletPicker (#14099) [#14103](https://github.com/vitessio/vitess/pull/14103) + * [release-17.0] VDiff: Cleanup the controller for a VDiff before deleting it (#14107) [#14125](https://github.com/vitessio/vitess/pull/14125) +### CI/Build +#### Documentation + * [release-17.0] update docgen to embed commit ID in autogenerated doc frontmatter (#14056) [#14074](https://github.com/vitessio/vitess/pull/14074) +#### General + * [release-17.0] Upgrade the Golang version to `go1.20.8` [#13934](https://github.com/vitessio/vitess/pull/13934) +#### VTorc + * [release-17.0] docker: add dedicated vtorc container (#14126) [#14147](https://github.com/vitessio/vitess/pull/14147) +### Documentation +#### Documentation + * [release-17.0] anonymize homedirs in generated docs (#14101) [#14106](https://github.com/vitessio/vitess/pull/14106) +### Enhancement +#### Build/CI + * [release-17.0] Bump upgrade tests to `go1.21.0` [#13855](https://github.com/vitessio/vitess/pull/13855) +### Internal Cleanup +#### Build/CI + * [release-17.0] Use Debian Bullseye in Bootstrap [#13757](https://github.com/vitessio/vitess/pull/13757) +#### Query Serving + * [release-17.0] moved timeout test to different package (#14028) [#14032](https://github.com/vitessio/vitess/pull/14032) +### Release +#### General + * Code freeze of release-17.0 [#14138](https://github.com/vitessio/vitess/pull/14138) +### Testing +#### Build/CI + * [release-17.0] Remove FOSSA Test from CI until we can do it in a secure way (#14119) [#14122](https://github.com/vitessio/vitess/pull/14122) + * [release-17.0] Fix upgrade tests [#14143](https://github.com/vitessio/vitess/pull/14143) +#### VReplication + * [release-17.0] Flakes: empty vtdataroot before starting a new vreplication e2e test (#13803) [#13822](https://github.com/vitessio/vitess/pull/13822) + diff --git a/changelog/17.0/17.0.3/release_notes.md b/changelog/17.0/17.0.3/release_notes.md new file mode 100644 index 00000000000..27c87fb94a9 --- /dev/null +++ b/changelog/17.0/17.0.3/release_notes.md @@ -0,0 +1,30 @@ +# Release of Vitess v17.0.3 +## Summary + +### Table of Contents + +- **[Major Changes](#major-changes)** + - **[New command line flags and behavior](#new-flag)** + - [VTGate flag `--grpc-send-session-in-streaming`](#new-vtgate-streaming-sesion) + +## Major Changes + +### New command line flags and behavior + +#### VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming` + +This flag enables transaction support on `StreamExecute` api. +One enabled, VTGate `StreamExecute` grpc api will send session as the last packet in the response. +The client should enable it only when they have made the required changes to expect such a packet. + +It is disabled by default from v17.0.3. + +This was a breaking change when v17.0.0 was released was causing upgrade issue for client +who relied on a certain behaviour of receiving streaming packets on `StreamExecute` call. +------------ +The entire changelog for this release can be found [here](https://github.com/vitessio/vitess/blob/main/changelog/17.0/17.0.3/changelog.md). + +The release includes 34 merged Pull Requests. + +Thanks to all our contributors: @ajm188, @app/github-actions, @app/vitess-bot, @frouioui, @mattlord, @rohit-nayak-ps + diff --git a/changelog/17.0/17.0.3/summary.md b/changelog/17.0/17.0.3/summary.md new file mode 100644 index 00000000000..39d87af9744 --- /dev/null +++ b/changelog/17.0/17.0.3/summary.md @@ -0,0 +1,22 @@ +## Summary + +### Table of Contents + +- **[Major Changes](#major-changes)** + - **[New command line flags and behavior](#new-flag)** + - [VTGate flag `--grpc-send-session-in-streaming`](#new-vtgate-streaming-sesion) + +## Major Changes + +### New command line flags and behavior + +#### VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming` + +This flag enables transaction support on `StreamExecute` api. +One enabled, VTGate `StreamExecute` grpc api will send session as the last packet in the response. +The client should enable it only when they have made the required changes to expect such a packet. + +It is disabled by default from v17.0.3. + +This was a breaking change when v17.0.0 was released was causing upgrade issue for client +who relied on a certain behaviour of receiving streaming packets on `StreamExecute` call. \ No newline at end of file diff --git a/changelog/17.0/README.md b/changelog/17.0/README.md index 641a52e1079..dcbba316bdd 100644 --- a/changelog/17.0/README.md +++ b/changelog/17.0/README.md @@ -1,4 +1,8 @@ ## v17.0 +* **[17.0.3](17.0.3)** + * [Changelog](17.0.3/changelog.md) + * [Release Notes](17.0.3/release_notes.md) + * **[17.0.2](17.0.2)** * [Changelog](17.0.2/changelog.md) * [Release Notes](17.0.2/release_notes.md) diff --git a/changelog/18.0/18.0.0/changelog.md b/changelog/18.0/18.0.0/changelog.md new file mode 100644 index 00000000000..8648ced0d02 --- /dev/null +++ b/changelog/18.0/18.0.0/changelog.md @@ -0,0 +1,464 @@ +# Changelog of Vitess v18.0.0-rc1 + +### Bug fixes +#### Backup and Restore +* vtctldclient: Add missing new backup option [#13543](https://github.com/vitessio/vitess/pull/13543) +* Backup: safe compressor/decompressor closure [#13668](https://github.com/vitessio/vitess/pull/13668) +* Address vttablet memory usage with backups to Azure Blob Service [#13770](https://github.com/vitessio/vitess/pull/13770) +* Do not drain tablet in incremental backup [#13773](https://github.com/vitessio/vitess/pull/13773) +* go/cmd/vtbackup: wait for plugins to finish initializing [#14113](https://github.com/vitessio/vitess/pull/14113) +#### Build/CI +* Remove `os.Exit` in release-notes generation [#13310](https://github.com/vitessio/vitess/pull/13310) +* CI: Fix make build related issues [#13583](https://github.com/vitessio/vitess/pull/13583) +* Enable failures in `tools/e2e_test_race.sh` and fix races [#13654](https://github.com/vitessio/vitess/pull/13654) +* Fix regular expression issue in Golang Upgrade and remove `release-14.0` from target branch [#13846](https://github.com/vitessio/vitess/pull/13846) +* Make `Static Code Checks Etc` fail if the `./changelog` folder is out-of-date [#14003](https://github.com/vitessio/vitess/pull/14003) +#### CLI +* viperutil: Remove potential cross site reflecting issue [#13483](https://github.com/vitessio/vitess/pull/13483) +* [vtctldclient] flags need to be defined to be deprecated [#13681](https://github.com/vitessio/vitess/pull/13681) +* Fix missing deprecated flags in `vttablet` and `vtgate` [#13975](https://github.com/vitessio/vitess/pull/13975) +#### Cluster management +* Prevent resetting replication every time we set replication source [#13377](https://github.com/vitessio/vitess/pull/13377) +* Don't run any reparent commands if the host is empty [#13396](https://github.com/vitessio/vitess/pull/13396) +* [main] Upgrade-Downgrade Fix: Schema-initialization stuck on semi-sync ACKs while upgrading (#13411) [#13440](https://github.com/vitessio/vitess/pull/13440) +* fix: error.as method usage to send pointer to the reference type expected [#13496](https://github.com/vitessio/vitess/pull/13496) +* check keyspace snapshot time if none specified for backup restores [#13557](https://github.com/vitessio/vitess/pull/13557) +* Flaky tests: Fix race in memory topo [#13559](https://github.com/vitessio/vitess/pull/13559) +* ignore all error for views in engine reload [#13590](https://github.com/vitessio/vitess/pull/13590) +* Fix `BackupShard` to get its options from its own flags [#13813](https://github.com/vitessio/vitess/pull/13813) +#### Docker +* Fix ubi8.arm64.mysql80 build package mirrorserver error [#13431](https://github.com/vitessio/vitess/pull/13431) +* Fix dependencies in docker build script [#13520](https://github.com/vitessio/vitess/pull/13520) +* Use NodeJS v18 in VTAdmin Dockerfile [#13751](https://github.com/vitessio/vitess/pull/13751) +#### Evalengine +* Fix a number of encoding issues when evaluating expressions with the evalengine [#13509](https://github.com/vitessio/vitess/pull/13509) +* Fix type comparisons for Nullsafe* functions [#13605](https://github.com/vitessio/vitess/pull/13605) +* fastparse: Fix bug in overflow detection [#13702](https://github.com/vitessio/vitess/pull/13702) +* evalengine: Mark UUID() function as non-constant [#14051](https://github.com/vitessio/vitess/pull/14051) +#### Examples +* Use $hostname in vtadmin script as all other scripts do [#13231](https://github.com/vitessio/vitess/pull/13231) +* Local example 101: idempotent on existing clusters [#13373](https://github.com/vitessio/vitess/pull/13373) +* Examples: only terminate vtadmin if it was started [#13433](https://github.com/vitessio/vitess/pull/13433) +* `examples/compose`: fix `consul:latest` error w/`docker-compose up -d` [#13468](https://github.com/vitessio/vitess/pull/13468) +#### General +* Flakes: Synchronize access to logErrStacks in vterrors [#13827](https://github.com/vitessio/vitess/pull/13827) +#### Online DDL +* Solve RevertMigration.Comment read/write concurrency issue [#13700](https://github.com/vitessio/vitess/pull/13700) +* Fix closed channel `panic` in Online DDL cutover [#13729](https://github.com/vitessio/vitess/pull/13729) +* OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations [#13928](https://github.com/vitessio/vitess/pull/13928) +* Fix `ApplySchema --batch-size` with ` --allow-zero-in-date` [#13951](https://github.com/vitessio/vitess/pull/13951) +* TableGC: support DROP VIEW [#14020](https://github.com/vitessio/vitess/pull/14020) +* OnlineDDL: cleanup cancelled migration artifacts; support `--retain-artifacts=` DDL strategy flag [#14029](https://github.com/vitessio/vitess/pull/14029) +* bugfix: change column name and type to json [#14093](https://github.com/vitessio/vitess/pull/14093) +#### Query Serving +* fix: GetField to use existing session for query [#13219](https://github.com/vitessio/vitess/pull/13219) +* VReplication Workflows: make sequence tables follow routing rules [#13238](https://github.com/vitessio/vitess/pull/13238) +* Adding random query generation for endtoend testing of the Gen4 planner [#13260](https://github.com/vitessio/vitess/pull/13260) +* Bug fix: SQL queries erroring with message `unknown aggregation random` [#13330](https://github.com/vitessio/vitess/pull/13330) +* bugfixes: collection of fixes to bugs found while fuzzing [#13332](https://github.com/vitessio/vitess/pull/13332) +* bug: don't always wrap aggregation in coalesce [#13348](https://github.com/vitessio/vitess/pull/13348) +* Random selection of keyspace based on available tablet [#13359](https://github.com/vitessio/vitess/pull/13359) +* Enable Tcp keep alive and provide keep alive period setting [#13434](https://github.com/vitessio/vitess/pull/13434) +* Improving random query generation for endtoend testing [#13460](https://github.com/vitessio/vitess/pull/13460) +* ignore ongoing backfill vindex from routing selection [#13505](https://github.com/vitessio/vitess/pull/13505) +* [release-17.0] ignore ongoing backfill vindex from routing selection [#13523](https://github.com/vitessio/vitess/pull/13523) +* Fix flaky vtgate test TestInconsistentStateDetectedBuffering [#13560](https://github.com/vitessio/vitess/pull/13560) +* Fix show character set [#13565](https://github.com/vitessio/vitess/pull/13565) +* vtgate: fix race condition iterating tables and views from schema tracker [#13673](https://github.com/vitessio/vitess/pull/13673) +* sqlparser: Track if original default value is a literal [#13730](https://github.com/vitessio/vitess/pull/13730) +* Fix for "text type with an unknown/unsupported collation cannot be hashed" error [#13852](https://github.com/vitessio/vitess/pull/13852) +* VTGate Buffering: Use a more accurate heuristic for determining if we're doing a reshard [#13856](https://github.com/vitessio/vitess/pull/13856) +* sqlparser: Tablespace option is case sensitive [#13884](https://github.com/vitessio/vitess/pull/13884) +* Rewrite `USING` to `ON` condition for joins [#13931](https://github.com/vitessio/vitess/pull/13931) +* handle large number of predicates without timing out [#13979](https://github.com/vitessio/vitess/pull/13979) +* Fix `NOT IN` expression used in the SET NULL for a child table on an update [#13988](https://github.com/vitessio/vitess/pull/13988) +* Fix the `SELECT` query we run on the child table to verify that update is allowed on a RESTRICT constraint [#13991](https://github.com/vitessio/vitess/pull/13991) +* fix data race in join engine primitive olap streaming mode execution [#14012](https://github.com/vitessio/vitess/pull/14012) +* fix: cost to include subshard opcode [#14023](https://github.com/vitessio/vitess/pull/14023) +* Add session flag for stream execute grpc api [#14046](https://github.com/vitessio/vitess/pull/14046) +* Fix cascading Delete failure while using Prepared statements [#14048](https://github.com/vitessio/vitess/pull/14048) +* Fix Fk verification and update queries to accommodate for bindVariables being NULL [#14061](https://github.com/vitessio/vitess/pull/14061) +* DDL execution to commit open transaction [#14110](https://github.com/vitessio/vitess/pull/14110) +#### Schema Tracker +* Vttablet schema tracking: Fix _vt.schema_version corruption [#13045](https://github.com/vitessio/vitess/pull/13045) +* Ignore error while reading table data in Schema.Engine reload [#13421](https://github.com/vitessio/vitess/pull/13421) +* schema.Reload(): ignore column reading errors for views only, error for tables [#13442](https://github.com/vitessio/vitess/pull/13442) +#### TabletManager +* mysqlctl: Correctly encode database and table names [#13312](https://github.com/vitessio/vitess/pull/13312) +* Fix remote VersionString API [#13484](https://github.com/vitessio/vitess/pull/13484) +* mysqlctl: Remove noisy log line [#13599](https://github.com/vitessio/vitess/pull/13599) +* GetSchema: limit concurrent operations [#13617](https://github.com/vitessio/vitess/pull/13617) +* mysqlctl: Reduce logging for running commands [#13659](https://github.com/vitessio/vitess/pull/13659) +#### Throttler +* Tablet throttler: only start watching SrvKeyspace once it's confirmed to exist [#13384](https://github.com/vitessio/vitess/pull/13384) +* Throttler: reintroduce deprecated flags so that deprecation actually works [#13597](https://github.com/vitessio/vitess/pull/13597) +* Silence 'CheckThrottler' gRPC calls [#13925](https://github.com/vitessio/vitess/pull/13925) +* Tablet throttler: empty list of probes on non-leader [#13926](https://github.com/vitessio/vitess/pull/13926) +#### VReplication +* VReplication: Fix VDiff2 DeleteByUUID Query [#13255](https://github.com/vitessio/vitess/pull/13255) +* Better handling of vreplication setState() failure [#13488](https://github.com/vitessio/vitess/pull/13488) +* VReplication: Ignore unrelated shards in partial MoveTables traffic state [#13515](https://github.com/vitessio/vitess/pull/13515) +* VReplication: Ensure ROW events are sent within a transaction [#13547](https://github.com/vitessio/vitess/pull/13547) +* VReplication: Make Source Tablet Selection More Robust [#13582](https://github.com/vitessio/vitess/pull/13582) +* vtgate tablet gateway buffering: don't shutdown if not initialized [#13695](https://github.com/vitessio/vitess/pull/13695) +* VReplication: Improve MoveTables Create Error Handling [#13737](https://github.com/vitessio/vitess/pull/13737) +* Minor --initialize-target-sequences followups [#13758](https://github.com/vitessio/vitess/pull/13758) +* Flakes: skip flaky check that ETA for a VReplication VDiff2 Progress command is in the future. [#13804](https://github.com/vitessio/vitess/pull/13804) +* Flakes: VReplication unit tests: reduce goroutine leakage [#13824](https://github.com/vitessio/vitess/pull/13824) +* Properly support ignore_nulls in CreateLookupVindex [#13913](https://github.com/vitessio/vitess/pull/13913) +* VReplication: Handle SQL NULL and JSON 'null' correctly for JSON columns [#13944](https://github.com/vitessio/vitess/pull/13944) +* copy over existing vreplication rows copied to local counter if resuming from another tablet [#13949](https://github.com/vitessio/vitess/pull/13949) +* VDiff: correct handling of default source and target cells [#13969](https://github.com/vitessio/vitess/pull/13969) +* MoveTables Cancel: drop denied tables on target when dropping source/target tables [#14008](https://github.com/vitessio/vitess/pull/14008) +* VReplication VPlayer: set foreign_key_checks on initialization [#14013](https://github.com/vitessio/vitess/pull/14013) +* json: Fix quoting JSON keys [#14066](https://github.com/vitessio/vitess/pull/14066) +* VDiff: properly split cell values in record when using TabletPicker [#14099](https://github.com/vitessio/vitess/pull/14099) +* VDiff: Cleanup the controller for a VDiff before deleting it [#14107](https://github.com/vitessio/vitess/pull/14107) +#### VTAdmin +* Unset the PREFIX environment variable when building VTAdmin [#13554](https://github.com/vitessio/vitess/pull/13554) +#### VTCombo +* Fix vtcombo DBDDL plugin race condition [#13117](https://github.com/vitessio/vitess/pull/13117) +#### VTorc +* Ensure to call `servenv.Init` when needed [#13638](https://github.com/vitessio/vitess/pull/13638) +### CI/Build +#### Backup and Restore +* Refactor `backup_pitr` into two distinct CI tests: builtin vs Xtrabackup [#13395](https://github.com/vitessio/vitess/pull/13395) +* Fixing `backup_pitr` flaky tests via wait-for loop on topo reads [#13781](https://github.com/vitessio/vitess/pull/13781) +#### Build/CI +* Update a number of dependencies [#13031](https://github.com/vitessio/vitess/pull/13031) +* Cleanup unused Dockerfile entries [#13327](https://github.com/vitessio/vitess/pull/13327) +* flags: Remove hardcoded runner paths [#13482](https://github.com/vitessio/vitess/pull/13482) +* added no-commit-collection option to launchable record build command [#13490](https://github.com/vitessio/vitess/pull/13490) +* Replace deprecated `github.com/golang/mock` with `go.uber.org/mock` [#13512](https://github.com/vitessio/vitess/pull/13512) +* [viper WatchConfig] platform-specific write to ensure callback fires exactly once [#13627](https://github.com/vitessio/vitess/pull/13627) +* build: Allow passing in custom -ldflags [#13748](https://github.com/vitessio/vitess/pull/13748) +* Run auto golang upgrade only on vitessio/vitess [#13766](https://github.com/vitessio/vitess/pull/13766) +* collations: implement collation dumping as a docker image [#13879](https://github.com/vitessio/vitess/pull/13879) +#### Docker +* docker/k8s: add bookworm builds [#13436](https://github.com/vitessio/vitess/pull/13436) +* Bump docker images to `bullseye` [#13664](https://github.com/vitessio/vitess/pull/13664) +#### Documentation +* fix docgen for subcommands [#13518](https://github.com/vitessio/vitess/pull/13518) +* update docgen to embed commit ID in autogenerated doc frontmatter [#14056](https://github.com/vitessio/vitess/pull/14056) +#### General +* go/mysql: switch to new API for x/exp/slices.SortFunc [#13644](https://github.com/vitessio/vitess/pull/13644) +* [main] Upgrade the Golang version to `go1.21.1` [#13933](https://github.com/vitessio/vitess/pull/13933) +#### Online DDL +* CI: fix onlineddl_scheduler flakiness [#13754](https://github.com/vitessio/vitess/pull/13754) +#### Query Serving +* Endtoend: stress tests for VTGate FOREIGN KEY support [#13799](https://github.com/vitessio/vitess/pull/13799) +* ci: pool-related test flakyness [#14076](https://github.com/vitessio/vitess/pull/14076) +#### Throttler +* Deprecating and removing tablet throttler CLI flags and tests [#13246](https://github.com/vitessio/vitess/pull/13246) +* Throttler: verify deprecated flags are still allowed [#13615](https://github.com/vitessio/vitess/pull/13615) +#### VReplication +* Flakes: Remove CI endtoend test for VReplication Copy Phase Throttling [#13343](https://github.com/vitessio/vitess/pull/13343) +* Flakes: Improve reliability of vreplication_copy_parallel test [#13857](https://github.com/vitessio/vitess/pull/13857) +#### VTAdmin +* Improve time taken to run the examples by optimizing `vtadmin` build [#13262](https://github.com/vitessio/vitess/pull/13262) +#### VTorc +* [release-18.0] docker: add dedicated vtorc container (#14126) [#14148](https://github.com/vitessio/vitess/pull/14148) +### Dependabot +#### General +* Bump word-wrap from 1.2.3 to 1.2.4 in /web/vtadmin [#13569](https://github.com/vitessio/vitess/pull/13569) +* Bump tough-cookie from 4.1.2 to 4.1.3 in /web/vtadmin [#13767](https://github.com/vitessio/vitess/pull/13767) +#### Java +* java: update to latest dependencies for grpc and protobuf [#13996](https://github.com/vitessio/vitess/pull/13996) +#### Observability +* Bump tough-cookie and @cypress/request in /vitess-mixin/e2e [#13768](https://github.com/vitessio/vitess/pull/13768) +#### VTAdmin +* build(deps-dev): bump vite from 4.2.1 to 4.2.3 in /web/vtadmin [#13240](https://github.com/vitessio/vitess/pull/13240) +* Bump protobufjs from 7.2.3 to 7.2.5 in /web/vtadmin [#13833](https://github.com/vitessio/vitess/pull/13833) +### Documentation +#### CLI +* gentler warning message on config-not-found [#13215](https://github.com/vitessio/vitess/pull/13215) +* switch casing in onlineddl subcommand help text [#14091](https://github.com/vitessio/vitess/pull/14091) +#### Documentation +* Add security audit report [#13221](https://github.com/vitessio/vitess/pull/13221) +* update link for reparenting guide [#13350](https://github.com/vitessio/vitess/pull/13350) +* anonymize homedirs in generated docs [#14101](https://github.com/vitessio/vitess/pull/14101) +* Summary changes for foreign keys [#14112](https://github.com/vitessio/vitess/pull/14112) +* fix bad copy-paste in zkctld docgen [#14123](https://github.com/vitessio/vitess/pull/14123) +#### General +* Improve release process documentation [#14000](https://github.com/vitessio/vitess/pull/14000) +#### Governance +* governance doc clean up [#13337](https://github.com/vitessio/vitess/pull/13337) +### Enhancement +#### Backup and Restore +* go/vt/mysqlctl: instrument s3 upload time [#12500](https://github.com/vitessio/vitess/pull/12500) +* metrics: change vtbackup_duration_by_phase to binary-valued vtbackup_phase [#12973](https://github.com/vitessio/vitess/pull/12973) +* Incremental backup & recovery: restore-to-timestamp [#13270](https://github.com/vitessio/vitess/pull/13270) +* backup: Allow for upgrade safe backups [#13449](https://github.com/vitessio/vitess/pull/13449) +* Incremental backup: accept GTID position without 'MySQL56/' flavor prefix [#13474](https://github.com/vitessio/vitess/pull/13474) +* Backup & Restore: vtctldclient to support PITR flags [#13513](https://github.com/vitessio/vitess/pull/13513) +* BackupShard: support incremental backup [#13522](https://github.com/vitessio/vitess/pull/13522) +* Point in time recovery: fix cross-tablet GTID evaluation [#13555](https://github.com/vitessio/vitess/pull/13555) +* Backup/restore: provision and restore a tablet with point-in-time recovery flags [#13964](https://github.com/vitessio/vitess/pull/13964) +* go/cmd/vtbackup: report replication status metrics during catch-up phase [#13995](https://github.com/vitessio/vitess/pull/13995) +#### Build/CI +* Set the number of threads for release notes generation with a flag [#13273](https://github.com/vitessio/vitess/pull/13273) +* Optimize `make build` in `test.go` and in CI [#13567](https://github.com/vitessio/vitess/pull/13567) +* Skip VTAdmin build in more places [#13588](https://github.com/vitessio/vitess/pull/13588) +* Skip VTAdmin build in Docker tests [#13836](https://github.com/vitessio/vitess/pull/13836) +* Migrates most workflows to 4 and 16 cores Large GitHub-Hosted-Runners [#13845](https://github.com/vitessio/vitess/pull/13845) +* Skip launchable if the Pull Request is marked as a Draft [#13886](https://github.com/vitessio/vitess/pull/13886) +#### CLI +* Vtctldclient MoveTables [#13015](https://github.com/vitessio/vitess/pull/13015) +* migrate vtorc to use cobra commands [#13917](https://github.com/vitessio/vitess/pull/13917) +#### Cluster management +* increase length of reparent_journal columns [#13287](https://github.com/vitessio/vitess/pull/13287) +* Improvements to PRS [#13623](https://github.com/vitessio/vitess/pull/13623) +* Add 2 more durability policies that allow RDONLY tablets to send semi-sync ACKs [#13698](https://github.com/vitessio/vitess/pull/13698) +* `vtctld`/`vtorc`: improve reparenting stats [#13723](https://github.com/vitessio/vitess/pull/13723) +#### Documentation +* consolidate docs [#13959](https://github.com/vitessio/vitess/pull/13959) +#### Evalengine +* evalengine: implement date/time math [#13274](https://github.com/vitessio/vitess/pull/13274) +* sqlparser: Add support for TIMESTAMPADD [#13314](https://github.com/vitessio/vitess/pull/13314) +* mysql: introduce icuregex package [#13391](https://github.com/vitessio/vitess/pull/13391) +* icuregex: Lazy load ICU data into memory [#13640](https://github.com/vitessio/vitess/pull/13640) +* evalengine: Improve weight string support [#13658](https://github.com/vitessio/vitess/pull/13658) +* evalengine: Fix JSON weight string computation [#13669](https://github.com/vitessio/vitess/pull/13669) +#### Examples +* Misc Local Install improvements. [#13446](https://github.com/vitessio/vitess/pull/13446) +#### General +* Refactor code to remove `evalengine` as a dependency of `VTOrc` [#13642](https://github.com/vitessio/vitess/pull/13642) +#### Observability +* vtorc: add detected_problems counter [#13967](https://github.com/vitessio/vitess/pull/13967) +#### Online DDL +* `vtctl OnlineDDL`: complete command set [#12963](https://github.com/vitessio/vitess/pull/12963) +* Online DDL: improved row estimation via ANALYE TABLE with --analyze-table strategy flag [#13352](https://github.com/vitessio/vitess/pull/13352) +* OnlineDDL: support @@migration_context in vtgate session. Use if non-empty [#13675](https://github.com/vitessio/vitess/pull/13675) +* Vtgate: pass 'SHOW VITESS_MIGRATIONS' to tablet's query executor [#13726](https://github.com/vitessio/vitess/pull/13726) +* vtctldclient OnlineDDL CANCEL [#13860](https://github.com/vitessio/vitess/pull/13860) +* vtctldclient: support OnlineDDL `complete`, `launch` commands [#13896](https://github.com/vitessio/vitess/pull/13896) +#### Query Serving +* vindexes: return unknown params [#12951](https://github.com/vitessio/vitess/pull/12951) +* Fix and Make aggregation planner handle aggregation functions better [#13228](https://github.com/vitessio/vitess/pull/13228) +* vtgate planner: HAVING in the new operator horizon planner [#13289](https://github.com/vitessio/vitess/pull/13289) +* Support complex aggregation in Gen4's Operators [#13326](https://github.com/vitessio/vitess/pull/13326) +* Adds support for ANY_VALUE [#13342](https://github.com/vitessio/vitess/pull/13342) +* Aggregation engine refactor [#13378](https://github.com/vitessio/vitess/pull/13378) +* Move more horizon planning to the operators [#13412](https://github.com/vitessio/vitess/pull/13412) +* Move UNION planning to the operators [#13450](https://github.com/vitessio/vitess/pull/13450) +* Improve and Fix Distinct Aggregation planner [#13466](https://github.com/vitessio/vitess/pull/13466) +* Enhancing VTGate buffering for MoveTables and Shard by Shard Migration [#13507](https://github.com/vitessio/vitess/pull/13507) +* Add 2 new metrics with tablet type labels [#13521](https://github.com/vitessio/vitess/pull/13521) +* vtgate table schema tracking to use GetSchema rpc [#13544](https://github.com/vitessio/vitess/pull/13544) +* Add a `keyspace` configuration in the `vschema` for foreign key mode [#13553](https://github.com/vitessio/vitess/pull/13553) +* Reduce usages of old horizon planning fallback [#13595](https://github.com/vitessio/vitess/pull/13595) +* Add dry-run/monitoring-only mode for TxThrottler [#13604](https://github.com/vitessio/vitess/pull/13604) +* go/vt/vitessdriver: implement driver.{Connector,DriverContext} [#13704](https://github.com/vitessio/vitess/pull/13704) +* More union merging [#13743](https://github.com/vitessio/vitess/pull/13743) +* Move subqueries to use the operator model [#13750](https://github.com/vitessio/vitess/pull/13750) +* Add support for tuple as value type [#13800](https://github.com/vitessio/vitess/pull/13800) +* icuregex: Update to ICU 73 [#13912](https://github.com/vitessio/vitess/pull/13912) +* Change internal vindex type recommendation for integrals to xxhash [#13956](https://github.com/vitessio/vitess/pull/13956) +* Foreign key cascade: retain "for update" lock on select query plans [#13985](https://github.com/vitessio/vitess/pull/13985) +* Improve the rewriter to simplify more queries [#14059](https://github.com/vitessio/vitess/pull/14059) +* [release-18.0] gen4: Support explicit column aliases on derived tables (#14129) [#14156](https://github.com/vitessio/vitess/pull/14156) +#### Schema Tracker +* vttablet: do not notify `vtgate` about internal tables [#13897](https://github.com/vitessio/vitess/pull/13897) +#### TabletManager +* Tablet throttler: throttled app configuration via `vtctl UpdateThrottlerConfig` [#13351](https://github.com/vitessio/vitess/pull/13351) +#### Throttler +* txthrottler: verify config at vttablet startup, consolidate funcs [#13115](https://github.com/vitessio/vitess/pull/13115) +* txthrottler: add metrics for topoWatcher and healthCheckStreamer [#13153](https://github.com/vitessio/vitess/pull/13153) +* `UpdateThrottlerConfig --unthrottle-app ...` [#13494](https://github.com/vitessio/vitess/pull/13494) +* Reroute 'ALTER VITESS_MIGRATION ... THROTTLE ...' through topo [#13511](https://github.com/vitessio/vitess/pull/13511) +* Tablet throttler: inter-checks via gRPC [#13514](https://github.com/vitessio/vitess/pull/13514) +* Per workload TxThrottler metrics [#13526](https://github.com/vitessio/vitess/pull/13526) +* Throttler: exempt apps via `UpdateThrottlerConfig --throttle-app-exempt` [#13666](https://github.com/vitessio/vitess/pull/13666) +#### Topology +* Support arbitrary ZooKeeper config lines [#13829](https://github.com/vitessio/vitess/pull/13829) +#### VReplication +* MoveTables: allow copying all tables in a single atomic copy phase cycle [#13137](https://github.com/vitessio/vitess/pull/13137) +* VReplication: More intelligently manage vschema table entries on unsharded targets [#13220](https://github.com/vitessio/vitess/pull/13220) +* MoveTables sequence e2e tests: change terminology to use basic vs simple everywhere for partial movetables workflows [#13435](https://github.com/vitessio/vitess/pull/13435) +* wrangler,workflow/workflow: materialize from intersecting source shards based on primary vindexes [#13782](https://github.com/vitessio/vitess/pull/13782) +* Implement Reshard in vtctldclient [#13792](https://github.com/vitessio/vitess/pull/13792) +* VDiff: Migrate client command to vtctldclient [#13976](https://github.com/vitessio/vitess/pull/13976) +#### VTCombo +* `vttestserver`: persist vschema changes in `--persistent_mode` [#13065](https://github.com/vitessio/vitess/pull/13065) +#### VTorc +* Improve VTOrc failure detection to be able to better handle dead primary failures [#13190](https://github.com/vitessio/vitess/pull/13190) +* Add flag to VTOrc to enable/disable its ability to run ERS [#13259](https://github.com/vitessio/vitess/pull/13259) +* Add metric for showing the errant GTIDs in VTOrc [#13281](https://github.com/vitessio/vitess/pull/13281) +* Add timestamp to vtorc debug page [#13379](https://github.com/vitessio/vitess/pull/13379) +* Augment VTOrc to also store the shard records and use it to better judge Primary recoveries [#13587](https://github.com/vitessio/vitess/pull/13587) +* Fix a couple of logs in VTOrc [#13667](https://github.com/vitessio/vitess/pull/13667) +* Errant GTID Metrics Refactor [#13670](https://github.com/vitessio/vitess/pull/13670) +* VTOrc converts a tablet to DRAINED type if it detects errant GTIDs on it [#13873](https://github.com/vitessio/vitess/pull/13873) +#### vtctl +* vtctl,vindexes: logs warnings and export stat for unknown vindex params [#13322](https://github.com/vitessio/vitess/pull/13322) +* vtctldclient OnlineDDL: support `throttle`, `unthrottle` [#13916](https://github.com/vitessio/vitess/pull/13916) +#### web UI +* Add vtsql flags to vtadmin [#13674](https://github.com/vitessio/vitess/pull/13674) +### Feature Request +#### CLI +* [vtctld] more cobra binaries [#13930](https://github.com/vitessio/vitess/pull/13930) +* [cobra] vtgate and vttablet [#13943](https://github.com/vitessio/vitess/pull/13943) +* [cli] migrate mysqlctl and mysqlctld to cobra [#13946](https://github.com/vitessio/vitess/pull/13946) +* [CLI] cobra lots of things [#14007](https://github.com/vitessio/vitess/pull/14007) +* miscellaneous cobras [#14069](https://github.com/vitessio/vitess/pull/14069) +* [cli] cobra zookeeper [#14094](https://github.com/vitessio/vitess/pull/14094) +#### Online DDL +* Add OnlineDDL show support [#13738](https://github.com/vitessio/vitess/pull/13738) +* [onlineddl] retry and cleanup [#13830](https://github.com/vitessio/vitess/pull/13830) +#### Query Serving +* Add group_concat aggregation support [#13331](https://github.com/vitessio/vitess/pull/13331) +* Add support for kill statement [#13371](https://github.com/vitessio/vitess/pull/13371) +* Build foreign key definition in schema tracker [#13657](https://github.com/vitessio/vitess/pull/13657) +* Foreign Keys: `INSERT` planning [#13676](https://github.com/vitessio/vitess/pull/13676) +* Foreign Keys: `DELETE` planning [#13746](https://github.com/vitessio/vitess/pull/13746) +* Foreign Keys: `UPDATE` planning [#13762](https://github.com/vitessio/vitess/pull/13762) +* Add Foreign key Cascade engine primitive [#13802](https://github.com/vitessio/vitess/pull/13802) +* Foreign key cascade planning for DELETE and UPDATE queries [#13823](https://github.com/vitessio/vitess/pull/13823) +* Add Foreign key verify constraint engine primitive [#13848](https://github.com/vitessio/vitess/pull/13848) +* Add VSchema DDL support for dropping sequence and auto increment [#13882](https://github.com/vitessio/vitess/pull/13882) +* Update Cascade Planning leading to Foreign key constraint verification [#13902](https://github.com/vitessio/vitess/pull/13902) +* Disallow Insert with Duplicate key update and Replace Into queries on foreign key column, set locks on fk queries [#13953](https://github.com/vitessio/vitess/pull/13953) +#### VReplication +* VReplication: Initialize Sequence Tables Used By Tables Being Moved [#13656](https://github.com/vitessio/vitess/pull/13656) +* MoveTables: add flag to specify that routing rules should not be created when a movetables workflow is created [#13895](https://github.com/vitessio/vitess/pull/13895) +### Internal Cleanup +#### Build/CI +* docker/k8s: Cleanup done TODO [#13347](https://github.com/vitessio/vitess/pull/13347) +* Remove unused chromedriver [#13573](https://github.com/vitessio/vitess/pull/13573) +* docker/bootstrap: remove --no-cache flag [#13785](https://github.com/vitessio/vitess/pull/13785) +#### CLI +* remove query_analyzer binary and release [#14055](https://github.com/vitessio/vitess/pull/14055) +#### Cluster management +* Fix logging by omitting the host and port in `SetReadOnly` [#13470](https://github.com/vitessio/vitess/pull/13470) +* Improve logging and renaming PrimaryTermStartTimestamp in vttablets [#13625](https://github.com/vitessio/vitess/pull/13625) +#### Evalengine +* collations: Refactor to separate basic collation information from data [#13868](https://github.com/vitessio/vitess/pull/13868) +#### Examples +* docker/mini: remove refs to orc configs [#13495](https://github.com/vitessio/vitess/pull/13495) +#### General +* servenv: Allow for explicit bind address [#13188](https://github.com/vitessio/vitess/pull/13188) +* Remove `out.txt` and add `release-17.0` to go upgrade automation [#13261](https://github.com/vitessio/vitess/pull/13261) +* Deprecate VTGR [#13301](https://github.com/vitessio/vitess/pull/13301) +* mysql: Refactor dependencies [#13688](https://github.com/vitessio/vitess/pull/13688) +* Remove explicit usage of etcd v2 (api and storage) [#13791](https://github.com/vitessio/vitess/pull/13791) +* Go 1.21 cleanups [#13862](https://github.com/vitessio/vitess/pull/13862) +* [wrangler] cleanup unused functions [#13867](https://github.com/vitessio/vitess/pull/13867) +* [misc] Delete more unused functions, tidy up dupe imports [#13878](https://github.com/vitessio/vitess/pull/13878) +* Clean up deprecated slice header usage and unused code [#13880](https://github.com/vitessio/vitess/pull/13880) +* [misc] tidy imports [#13885](https://github.com/vitessio/vitess/pull/13885) +* [staticcheck] miscellaneous tidying [#13892](https://github.com/vitessio/vitess/pull/13892) +* [staticcheck] Cleanup deprecations [#13898](https://github.com/vitessio/vitess/pull/13898) +* Consolidate helper functions for working with proto3 time messages [#13905](https://github.com/vitessio/vitess/pull/13905) +* [staticcheck] Last few staticchecks! [#13909](https://github.com/vitessio/vitess/pull/13909) +* Remove deprecated flags before `v18.0.0` [#14071](https://github.com/vitessio/vitess/pull/14071) +#### Observability +* stats: use *time.Ticker instead of time.After() [#13492](https://github.com/vitessio/vitess/pull/13492) +#### Query Serving +* Operator planner refactor [#13294](https://github.com/vitessio/vitess/pull/13294) +* Refactor and add a comment to schema initialisation code [#13309](https://github.com/vitessio/vitess/pull/13309) +* vtgate v3 planner removal [#13458](https://github.com/vitessio/vitess/pull/13458) +* vtgate buffering logic: remove the deprecated healthcheck based implementation [#13584](https://github.com/vitessio/vitess/pull/13584) +* Refactor Expression and Statement Simplifier [#13636](https://github.com/vitessio/vitess/pull/13636) +* Remove duplicate ACL check in tabletserver handleHTTPConsolidations [#13876](https://github.com/vitessio/vitess/pull/13876) +* inputs method to return additional information about the input primitive [#13883](https://github.com/vitessio/vitess/pull/13883) +* refactor: move DML logic to sql_builder.go [#13920](https://github.com/vitessio/vitess/pull/13920) +* Fix `TestLeftJoinUsingUnsharded` and remove instability when running E2E locally [#13973](https://github.com/vitessio/vitess/pull/13973) +* Remove excessive logging in transactions [#14021](https://github.com/vitessio/vitess/pull/14021) +* moved timeout test to different package [#14028](https://github.com/vitessio/vitess/pull/14028) +#### TabletManager +* mysqlctl: Use DBA connection for schema operations [#13178](https://github.com/vitessio/vitess/pull/13178) +* k8stopo: Include deprecation warning [#13299](https://github.com/vitessio/vitess/pull/13299) +* k8stopo: Remove the deprecated Kubernetes topo [#13303](https://github.com/vitessio/vitess/pull/13303) +* vtgr: Remove deprecated vtgr [#13308](https://github.com/vitessio/vitess/pull/13308) +* mysqlctl: Move more to use built in MySQL client [#13338](https://github.com/vitessio/vitess/pull/13338) +#### Throttler +* `txthrottler`: remove `txThrottlerConfig` struct, rely on `tabletenv` [#13624](https://github.com/vitessio/vitess/pull/13624) +#### VReplication +* Use sqlparser for all dynamic query building in VDiff2 [#13319](https://github.com/vitessio/vitess/pull/13319) +* vreplication: Move to use collations package [#13566](https://github.com/vitessio/vitess/pull/13566) +#### VTAdmin +* [VTAdmin] Upgrade to use node 18.16.0 [#13288](https://github.com/vitessio/vitess/pull/13288) +#### VTorc +* VTOrc: Update the primary key for all the tables from `hostname, port` to `alias` [#13243](https://github.com/vitessio/vitess/pull/13243) +* vtorc: Cleanup more unused code [#13354](https://github.com/vitessio/vitess/pull/13354) +* Improve lock action string [#13355](https://github.com/vitessio/vitess/pull/13355) +* Improve VTOrc logging statements, now that we have alias as a field [#13428](https://github.com/vitessio/vitess/pull/13428) +* Remove excessive logging in VTOrc APIs [#13459](https://github.com/vitessio/vitess/pull/13459) +* [release-16.0] Remove excessive logging in VTOrc APIs (#13459) [#13462](https://github.com/vitessio/vitess/pull/13462) +#### vtexplain +* vtexplain: Fix passing through context for cleanup [#13900](https://github.com/vitessio/vitess/pull/13900) +### Performance +#### General +* proto: Faster clone [#13914](https://github.com/vitessio/vitess/pull/13914) +#### Query Serving +* Cache info schema table info [#13724](https://github.com/vitessio/vitess/pull/13724) +* gen4: Fast aggregations [#13904](https://github.com/vitessio/vitess/pull/13904) +* Cache v3 [#13939](https://github.com/vitessio/vitess/pull/13939) +* Reduce network pressure on multi row insert [#14064](https://github.com/vitessio/vitess/pull/14064) +* VTGate FK stress tests suite: improvements [#14098](https://github.com/vitessio/vitess/pull/14098) +#### TabletManager +* BaseShowTablesWithSizes: optimize MySQL 8.0 query [#13375](https://github.com/vitessio/vitess/pull/13375) +* Support views in BaseShowTablesWithSizes for MySQL 8.0 [#13394](https://github.com/vitessio/vitess/pull/13394) +#### vtctl +* `ApplySchema`: support `--batch-size` flag in 'direct' strategy [#13693](https://github.com/vitessio/vitess/pull/13693) +### Regression +#### Query Serving +* fix: ShardedRouting clone to clone slice of reference correctly [#13265](https://github.com/vitessio/vitess/pull/13265) +* Handle inconsistent state error in query buffering [#13333](https://github.com/vitessio/vitess/pull/13333) +### Release +#### Build/CI +* Fix incorrect output in release scripts [#13385](https://github.com/vitessio/vitess/pull/13385) +* Optimize release notes generation to use GitHub Milestones [#13398](https://github.com/vitessio/vitess/pull/13398) +#### Documentation +* Add end-of-life documentation + re-organize internal documentation [#13401](https://github.com/vitessio/vitess/pull/13401) +* Update known issues in `v16.x` and `v17.0.0` [#13618](https://github.com/vitessio/vitess/pull/13618) +#### General +* Copy v17.0.0-rc changelog to main [#13248](https://github.com/vitessio/vitess/pull/13248) +* Update release notes for 17.0.0-rc2 [#13306](https://github.com/vitessio/vitess/pull/13306) +* Forward port of release notes changes from v17.0.0 GA [#13370](https://github.com/vitessio/vitess/pull/13370) +* Add v15.0.4, v16.0.3, and v17.0.1 changelogs [#13661](https://github.com/vitessio/vitess/pull/13661) +* Copy release notes for v17.0.2 and v16.0.4 [#13811](https://github.com/vitessio/vitess/pull/13811) +* Code freeze of release-18.0 [#14131](https://github.com/vitessio/vitess/pull/14131) +### Testing +#### Build/CI +* Flakes: Address TestMigrate Failures [#12866](https://github.com/vitessio/vitess/pull/12866) +* [vipersync] skip flaky test [#13501](https://github.com/vitessio/vitess/pull/13501) +* [vipersync] deflake TestWatchConfig [#13545](https://github.com/vitessio/vitess/pull/13545) +* Fix bug in `fileNameFromPosition` test helper [#13778](https://github.com/vitessio/vitess/pull/13778) +* Flakes: Delete VTDATAROOT files in reparent test teardown within CI [#13793](https://github.com/vitessio/vitess/pull/13793) +* CI: Misc test improvements to limit failures with various runners [#13825](https://github.com/vitessio/vitess/pull/13825) +* actually test vtcombo [#14095](https://github.com/vitessio/vitess/pull/14095) +* Remove FOSSA Test from CI until we can do it in a secure way [#14119](https://github.com/vitessio/vitess/pull/14119) +#### Cluster management +* Fix `Fakemysqldaemon` to store the host and port after `SetReplicationSource` call [#13439](https://github.com/vitessio/vitess/pull/13439) +* Deflake `TestPlannedReparentShardPromoteReplicaFail` [#13548](https://github.com/vitessio/vitess/pull/13548) +* Flaky tests: Fix wrangler tests [#13568](https://github.com/vitessio/vitess/pull/13568) +#### General +* [CI] deflake viper sync tests [#13185](https://github.com/vitessio/vitess/pull/13185) +* Remove `--disable_active_reparents` flag in vttablet-up.sh [#13504](https://github.com/vitessio/vitess/pull/13504) +* Add leak checking for vtgate tests [#13835](https://github.com/vitessio/vitess/pull/13835) +#### Online DDL +* Fix potential panics due to "Fail in goroutine after test completed" [#13596](https://github.com/vitessio/vitess/pull/13596) +* [OnlineDDL] add label so break works as intended [#13691](https://github.com/vitessio/vitess/pull/13691) +#### Query Serving +* Deflake `TestQueryTimeoutWithDual` test [#13405](https://github.com/vitessio/vitess/pull/13405) +* Fix `TestGatewayBufferingWhileReparenting` flakiness [#13469](https://github.com/vitessio/vitess/pull/13469) +* fix TestQueryTimeoutWithTables flaky test [#13579](https://github.com/vitessio/vitess/pull/13579) +* schemadiff: add time measure test for massive schema load and diff [#13697](https://github.com/vitessio/vitess/pull/13697) +* End to end testing suite for foreign keys [#13870](https://github.com/vitessio/vitess/pull/13870) +* Fix setup order to avoid races [#13871](https://github.com/vitessio/vitess/pull/13871) +* Use correct syntax in test [#13907](https://github.com/vitessio/vitess/pull/13907) +* test: added test to check binlogs to contain the cascade events [#13970](https://github.com/vitessio/vitess/pull/13970) +* E2E Fuzzing testing for foreign keys [#13980](https://github.com/vitessio/vitess/pull/13980) +* Fix foreign key plan tests expectation [#13997](https://github.com/vitessio/vitess/pull/13997) +#### VReplication +* Flakes: remove non-determinism from vtctldclient MoveTables unit test [#13765](https://github.com/vitessio/vitess/pull/13765) +* Flakes: empty vtdataroot before starting a new vreplication e2e test [#13803](https://github.com/vitessio/vitess/pull/13803) +* Flakes: Add recently added 'select rows_copied' query to ignore list [#13993](https://github.com/vitessio/vitess/pull/13993) +#### VTorc +* Fix flakiness in `TestDeadPrimaryRecoversImmediately` [#13232](https://github.com/vitessio/vitess/pull/13232) +* Fix flakiness in VTOrc tests [#13489](https://github.com/vitessio/vitess/pull/13489) +* Skip flaky test `TestReadOutdatedInstanceKeys` [#13561](https://github.com/vitessio/vitess/pull/13561) +* Reintroduce `TestReadOutdatedInstanceKeys` with debugging information [#13562](https://github.com/vitessio/vitess/pull/13562) +#### vtctl +* Fix merge conflict with new tests [#13869](https://github.com/vitessio/vitess/pull/13869) diff --git a/changelog/18.0/18.0.0/release_notes.md b/changelog/18.0/18.0.0/release_notes.md new file mode 100644 index 00000000000..c48ff018c85 --- /dev/null +++ b/changelog/18.0/18.0.0/release_notes.md @@ -0,0 +1,302 @@ +# Release of Vitess v18.0.0-rc1 +## Summary + +### Table of Contents + +- **[Major Changes](#major-changes)** + - **[Breaking Changes](#breaking-changes)** + - [Local examples now use etcd v3 storage and API](#local-examples-etcd-v3) + - **[New command line flags and behavior](#new-flag)** + - [VTOrc flag `--allow-emergency-reparent`](#new-flag-toggle-ers) + - [VTOrc flag `--change-tablets-with-errant-gtid-to-drained`](#new-flag-errant-gtid-convert) + - [ERS sub flag `--wait-for-all-tablets`](#new-ers-subflag) + - [VTGate flag `--grpc-send-session-in-streaming`](#new-vtgate-streaming-sesion) + - **[Experimental Foreign Key Support](#foreign-keys)** + - **[VTAdmin](#vtadmin)** + - [Updated to node v18.16.0](#update-node) + - **[Deprecations and Deletions](#deprecations-and-deletions)** + - [Deprecated Flags](#deprecated-flags) + - [Deprecated Stats](#deprecated-stats) + - [Deleted Flags](#deleted-flags) + - [Deleted `V3` planner](#deleted-v3) + - [Deleted `k8stopo`](#deleted-k8stopo) + - [Deleted `vtgr`](#deleted-vtgr) + - [Deleted `query_analyzer`](#deleted-query_analyzer) + - **[New Stats](#new-stats)** + - [VTGate Vindex unknown parameters](#vtgate-vindex-unknown-parameters) + - [VTBackup stat `Phase`](#vtbackup-stat-phase) + - [VTBackup stat `PhaseStatus`](#vtbackup-stat-phase-status) + - [Backup and restore metrics for AWS S3](#backup-restore-metrics-aws-s3) + - [VTCtld and VTOrc reparenting stats](#vtctld-and-vtorc-reparenting-stats) + - **[VTTablet](#vttablet)** + - [VTTablet: New ResetSequences RPC](#vttablet-new-rpc-reset-sequences) + - **[Docker](#docker)** + - [Debian: Bookworm added and made default](#debian-bookworm) + - [Debian: Buster removed](#debian-buster) + - **[Durability Policies](#durability-policies)** + - [New Durability Policies](#new-durability-policies) + +## Major Changes + +### Breaking Changes + +#### Local examples now use etcd v3 storage and API +In previous releases the [local examples](https://github.com/vitessio/vitess/tree/main/examples/local) were +explicitly using etcd v2 storage (`etcd --enable-v2=true`) and API (`ETCDCTL_API=2`) mode. We have now +removed this legacy etcd usage and instead use the new (default) etcd v3 storage and API. Please see +[PR #13791](https://github.com/vitessio/vitess/pull/13791) for additional info. If you are using the local +examples in any sort of long-term non-testing capacity, then you will need to explicitly use the v2 storage +and API mode or [migrate your existing data from v2 to v3](https://etcd.io/docs/v3.5/tutorials/how-to-migrate/). + +### New command line flags and behavior + +#### VTOrc flag `--allow-emergency-reparent` + +VTOrc has a new flag `--allow-emergency-reparent` that allows the users to toggle the ability of VTOrc to run emergency +reparent operations. Users that want VTOrc to fix the replication issues, but don't want it to run any reparents +should start using this flag. By default, VTOrc will be able to run `EmergencyReparentShard`. Users must specify the +flag to `false` to change the behaviour. + +#### VTOrc flag `--change-tablets-with-errant-gtid-to-drained` + +VTOrc has a new flag `--change-tablets-with-errant-gtid-to-drained` that allows users to choose whether VTOrc should change the +tablet type of tablets with errant GTIDs to `DRAINED`. By default, it is disabled. + +This feature allows users to configure VTOrc such that any tablet that encounters errant GTIDs is automatically taken out of the +serving graph. These tablets can then be inspected for what the errant GTIDs are, and once fixed, they can rejoin the cluster. + +#### ERS sub flag `--wait-for-all-tablets` + +Running `EmergencyReparentShard` from the vtctldclient has a new sub-flag `--wait-for-all-tablets` that makes `EmergencyReparentShard` wait +for a response from all the tablets. Originally `EmergencyReparentShard` was meant only to be run when a primary tablet is unreachable. +We have realized now that there are cases when the replication is broken but all the tablets are reachable. In these cases, it is advisable to +call `EmergencyReparentShard` with `--wait-for-all-tablets` so that it does not ignore one of the tablets. + +#### VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming` + +This flag enables transaction support on `StreamExecute` api. +Once enabled, VTGate `StreamExecute` gRPC api will send session as the last packet in the response. +The client should enable it only when they have made the required changes to expect such a packet. + +It is disabled by default. + +### Experimental Foreign Key Support + +A new field `foreignKeyMode` has been added to the VSchema. This field can be provided for each keyspace. The VTGate flag `--foreign_key_mode` has been deprecated in favour of this field. + +There are 3 foreign key modes now supported in Vitess - +1. `FK_UNMANAGED` - + This mode represents the default behaviour in Vitess, where it does not manage foreign keys column references. Users are responsible for configuring foreign keys in MySQL in such a way that related rows, as determined by foreign keys, reside within the same shard. +2. `FK_MANAGED` [EXPERIMENTAL] - + In this experimental mode, Vitess is fully aware of foreign key relationships and actively tracks foreign key constraints using the schema tracker. Vitess takes charge of handling DML operations with foreign keys cascading updates, deletes and verifying restrict. It will also validate parent row existence. + This ensures that all the operations are logged in binary logs, unlike MySQL implementation of foreign keys. + This enables seamless integration of VReplication with foreign keys. + For more details on what operations Vitess takes please refer to the [design document for foreign keys](https://github.com/vitessio/vitess/issues/12967). +3. `FK_DISALLOW` - + In this mode Vitess explicitly disallows any DDL statements that try to create a foreign key constraint. This mode is equivalent to running VTGate with the flag `--foreign_key_mode=disallow`. + +#### Upgrade process + +After upgrading from v17 to v18, the users should specify the correct foreign key mode for all their keyspaces in the VSchema using the new property. +Once this change has taken effect, the deprecated flag `--foreign_key_mode` can be dropped from all the VTGates. + +### VTAdmin + +#### vtadmin-web updated to node v18.16.0 (LTS) + +Building `vtadmin-web` now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed +in https://nodejs.org/en/blog/release/v18.0.0, but none apply to VTAdmin. Full details on v18.16.0 are listed +on https://nodejs.org/en/blog/release/v18.16.0. + +### Deprecations and Deletions + +#### Deprecated Command Line Flags + +Throttler related `vttablet` flags: + +- `--throttle_threshold` is deprecated and will be removed in `v19.0` +- `--throttle_metrics_query` is deprecated and will be removed in `v19.0` +- `--throttle_metrics_threshold` is deprecated and will be removed in `v19.0` +- `--throttle_check_as_check_self` is deprecated and will be removed in `v19.0` +- `--throttler-config-via-topo` is deprecated after assumed `true` in `v17.0`. It will be removed in a future version. + +Cache related `vttablet` flags: + +- `--queryserver-config-query-cache-lfu` is deprecated and will be removed in `v19.0`. The query cache always uses a LFU implementation now. +- `--queryserver-config-query-cache-size` is deprecated and will be removed in `v19.0`. This option only applied to LRU caches, which are now unsupported. + +Buffering related `vtgate` flags: + +- `--buffer_implementation` is deprecated and will be removed in `v19.0` + +Cache related `vtgate` flags: + +- `--gate_query_cache_lfu` is deprecated and will be removed in `v19.0`. The query cache always uses a LFU implementation now. +- `--gate_query_cache_size` is deprecated and will be removed in `v19.0`. This option only applied to LRU caches, which are now unsupported. + +VTGate flags: + +- `--schema_change_signal_user` is deprecated and will be removed in `v19.0` +- `--foreign_key_mode` is deprecated and will be removed in `v19.0`. For more detail read the [foreign keys](#foreign-keys) section. + +VDiff v1: + +[VDiff v2 was added in Vitess 15.0](https://vitess.io/blog/2022-11-22-vdiff-v2/) and marked as GA in 16.0. +The [legacy v1 client command](https://vitess.io/docs/18.0/reference/vreplication/vdiffv1/) is now deprecated in Vitess 18.0 and will be **removed** in 19.0. +Please switch all of your usage to the [new VDiff client](https://vitess.io/docs/18.0/reference/vreplication/vdiff/) command ASAP. + + +#### Deprecated Stats + +The following `EmergencyReparentShard` stats are deprecated in `v18.0` and will be removed in `v19.0`: +- `ers_counter` +- `ers_success_counter` +- `ers_failure_counter` + +These metrics are replaced by [new reparenting stats introduced in `v18.0`](#vtctld-and-vtorc-reparenting-stats). + +VTBackup stat `DurationByPhase` is deprecated. Use the binary-valued `Phase` stat instead. + +#### Deleted Command Line Flags + +Flags in `vtcombo`: +- `--vtctld_addr` + +Flags in `vtctldclient ApplySchema`: +- `--skip-preflight` + +Flags in `vtctl ApplySchema`: +- `--skip_preflight` + +Flags in `vtgate`: +- `--vtctld_addr` + +Flags in `vttablet`: +- `--vtctld_addr` +- `--use_super_read_only` +- `--disable-replication-manager` +- `--init_populate_metadata` +- `--queryserver-config-pool-prefill-parallelism` +- `--queryserver-config-stream-pool-prefill-parallelism` +- `--queryserver-config-transaction-pool-prefill-parallelism` +- `--queryserver-config-schema-change-signal-interval` +- `--enable-lag-throttler` + +Flags in `vtctld`: +- `--vtctld_show_topology_crud` +- `--durability_policy` + +Flags in `vtorc`: +- `--lock-shard-timeout` +- `--orc_web_dir` + +#### Deleted `v3` planner + +The `Gen4` planner has been the default planner since Vitess 14. The `v3` planner was deprecated in Vitess 15 and has now been removed in this release. + +#### Deleted `k8stopo` + +The `k8stopo` has been deprecated in Vitess 17, also see https://github.com/vitessio/vitess/issues/13298. With Vitess 18 +the `k8stopo` has been removed. + +#### Deleted `vtgr` + +The `vtgr` has been deprecated in Vitess 17, also see https://github.com/vitessio/vitess/issues/13300. With Vitess 18 `vtgr` has been removed. + +#### Deleted `query_analyzer` + +The undocumented `query_analyzer` binary has been removed in Vitess 18, see https://github.com/vitessio/vitess/issues/14054. + +### New stats + +#### VTGate Vindex unknown parameters + +The VTGate stat `VindexUnknownParameters` gauges unknown Vindex parameters found in the latest VSchema pulled from the topology. + +#### VTBackup `Phase` stat + +In v17, the `vtbackup` stat `DurationByPhase` stat was added measuring the time spent by `vtbackup` in each phase. This stat turned out to be awkward to use in production, and has been replaced in v18 by a binary-valued `Phase` stat. + +`Phase` reports a 1 (active) or a 0 (inactive) for each of the following phases: + +* `CatchupReplication` +* `InitialBackup` +* `RestoreLastBackup` +* `TakeNewBackup` + +To calculate how long `vtbackup` has spent in a given phase, sum the 1-valued data points over time and multiply by the data collection or reporting interval. For example, in Prometheus: + +``` +sum_over_time(vtbackup_phase{phase="TakeNewBackup"}) * +``` +#### VTBackup `PhaseStatus` stat + +`PhaseStatus` reports a 1 (active) or a 0 (inactive) for each of the following phases and statuses: + +* `CatchupReplication` phase has statuses `Stalled` and `Stopped`. + * `Stalled` is set to `1` when replication stops advancing. + * `Stopped` is set to `1` when replication stops before `vtbackup` catches up with the primary. + +#### Backup and restore metrics for AWS S3 + +Requests to AWS S3 are instrumented in backup and restore metrics. For example: + +``` +vtbackup_backup_count{component="BackupStorage",implementation="S3",operation="AWS:Request:Send"} 823 +vtbackup_backup_duration_nanoseconds{component="BackupStorage",implementation="S3",operation="AWS:Request:Send"} 1.33632421437e+11 +vtbackup_restore_count{component="BackupStorage",implementation="S3",operation="AWS:Request:Send"} 165 +vtbackup_restore_count{component="BackupStorage",implementation="S3",operation="AWS:Request:Send"} 165 +``` + +#### VTCtld and VTOrc reparenting stats + +New VTCtld and VTorc stats were added to measure frequency of reparents by keyspace/shard: +- `emergency_reparent_counts` - Number of times `EmergencyReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. +- `planned_reparent_counts` - Number of times `PlannedReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. + +Also, the `reparent_shard_operation_timings` stat was added to provide per-operation timings of reparent operations. + +### VTTablet + +#### New ResetSequences rpc + +A new VTTablet RPC `ResetSequences` has been added, which is being used by `MoveTables` and `Migrate` for workflows +where a `sequence` table is being moved (https://github.com/vitessio/vitess/pull/13238). This has an impact on the +Vitess upgrade process from an earlier version if you need to use such a workflow before the entire cluster is upgraded. + +Any MoveTables or Migrate workflow that moves a sequence table should only be run after all vitess components have been +upgraded, and no upgrade should be done while such a workflow is in progress. + +#### New Dry-run/monitoring-only mode for the transaction throttler + +A new CLI flag `--tx-throttler-dry-run` to set the Transaction Throttler to monitoring-only/dry-run mode has been added. +If the transaction throttler is enabled with `--enable-tx-throttler` and the new dry-run flag is also specified, the +tablet will not actually throttle any transactions; however, it will increase the counters for transactions throttled +(`vttablet_transaction_throttler_throttled`). This allows users to deploy the transaction throttler in production and +gain observability on how much throttling would take place, without actually throttling any requests. + +### Docker + +#### Bookworm added and made default + +Bookworm was released on 2023-06-10, and will be the new default base container for Docker builds. +Bullseye images will still be built and available as long as the OS build is current, tagged with the `-bullseye` suffix. + +#### Buster removed + +Buster LTS supports will stop in June 2024, and Vitess v18.0 will be supported through October 2024. +To prevent supporting a deprecated buster build for several months after June 2024, we are preemptively +removing Vitess support. + +### Durability Policies + +#### New Durability Policies + +2 new inbuilt durability policies have been added to Vitess in this release namely `semi_sync_with_rdonly_ack` and `cross_cell_with_rdonly_ack`. These policies are exactly like `semi_sync` and `cross_cell` respectively, and differ just in the part where the rdonly tablets can also send semi-sync ACKs. + +------------ +The entire changelog for this release can be found [here](https://github.com/vitessio/vitess/blob/main/changelog/18.0/18.0.0/changelog.md). + +The release includes 361 merged Pull Requests. + +Thanks to all our contributors: @GuptaManan100, @Juneezee, @adsr, @ajm188, @app/dependabot, @app/github-actions, @app/vitess-bot, @arvind-murty, @austenLacy, @brendar, @davidpiegza, @dbussink, @deepthi, @derekperkins, @ejortegau, @frouioui, @harshit-gangal, @hkdsun, @jfg956, @jspawar, @mattlord, @maxenglander, @mdlayher, @notfelineit, @olyazavr, @pbibra, @peterlyoo, @rafer, @rohit-nayak-ps, @shlomi-noach, @systay, @timvaillancourt, @vmg, @yields diff --git a/changelog/18.0/18.0.0/summary.md b/changelog/18.0/18.0.0/summary.md index dd004b99da6..fdaedd98e47 100644 --- a/changelog/18.0/18.0.0/summary.md +++ b/changelog/18.0/18.0.0/summary.md @@ -15,14 +15,12 @@ - [Updated to node v18.16.0](#update-node) - **[Deprecations and Deletions](#deprecations-and-deletions)** - [Deprecated Flags](#deprecated-flags) - - [Deleted Flags](#deleted-flags) - [Deprecated Stats](#deprecated-stats) + - [Deleted Flags](#deleted-flags) - [Deleted `V3` planner](#deleted-v3) - [Deleted `k8stopo`](#deleted-k8stopo) - [Deleted `vtgr`](#deleted-vtgr) - [Deleted `query_analyzer`](#deleted-query_analyzer) - - [Deprecated VTBackup stat `DurationByPhase`](#deprecated-vtbackup-stat-duration-by-phase) - - [Deprecated VDiff v1](#deprecated-vdiff-v1) - **[New Stats](#new-stats)** - [VTGate Vindex unknown parameters](#vtgate-vindex-unknown-parameters) - [VTBackup stat `Phase`](#vtbackup-stat-phase) @@ -54,36 +52,36 @@ and API mode or [migrate your existing data from v2 to v3](https://etcd.io/docs/ #### VTOrc flag `--allow-emergency-reparent` VTOrc has a new flag `--allow-emergency-reparent` that allows the users to toggle the ability of VTOrc to run emergency -reparent operations. The users that want VTOrc to fix the replication issues, but don't want it to run any reparents -should start using this flag. By default, VTOrc will be able to run `EmergencyReparentShard`. The users must specify the +reparent operations. Users that want VTOrc to fix the replication issues, but don't want it to run any reparents +should start using this flag. By default, VTOrc will be able to run `EmergencyReparentShard`. Users must specify the flag to `false` to change the behaviour. #### VTOrc flag `--change-tablets-with-errant-gtid-to-drained` VTOrc has a new flag `--change-tablets-with-errant-gtid-to-drained` that allows users to choose whether VTOrc should change the -tablet type of tablets with errant GTIDs to `DRAINED`. By default, the flag is false. +tablet type of tablets with errant GTIDs to `DRAINED`. By default, it is disabled. This feature allows users to configure VTOrc such that any tablet that encounters errant GTIDs is automatically taken out of the serving graph. These tablets can then be inspected for what the errant GTIDs are, and once fixed, they can rejoin the cluster. #### ERS sub flag `--wait-for-all-tablets` -Running `EmergencyReparentShard` from the vtctldclient has a new sub-flag `--wait-for-all-tablets` that makes `EmergencyReparentShard` wait +Running `EmergencyReparentShard` from the vtctldclient has a new sub-flag `--wait-for-all-tablets` that makes `EmergencyReparentShard` wait for a response from all the tablets. Originally `EmergencyReparentShard` was meant only to be run when a primary tablet is unreachable. -We have realized now that there are cases when the replication is broken but all the tablets are reachable. In these cases, it is advisable to -call `EmergencyReparentShard` with `--wait-for-all-tablets` so that it doesn't ignore one of the tablets. +We have realized now that there are cases when the replication is broken but all the tablets are reachable. In these cases, it is advisable to +call `EmergencyReparentShard` with `--wait-for-all-tablets` so that it does not ignore one of the tablets. #### VTGate GRPC stream execute session flag `--grpc-send-session-in-streaming` This flag enables transaction support on `StreamExecute` api. -One enabled, VTGate `StreamExecute` grpc api will send session as the last packet in the response. +Once enabled, VTGate `StreamExecute` gRPC api will send session as the last packet in the response. The client should enable it only when they have made the required changes to expect such a packet. It is disabled by default. ### Experimental Foreign Key Support -A new field `foreignKeyMode` has been added to the Vschema. This field can be provided for each keyspace. The Vtgate flag `--foreign_key_mode` has been deprecated in favour of this field. +A new field `foreignKeyMode` has been added to the VSchema. This field can be provided for each keyspace. The VTGate flag `--foreign_key_mode` has been deprecated in favour of this field. There are 3 foreign key modes now supported in Vitess - 1. `FK_UNMANAGED` - @@ -94,19 +92,20 @@ There are 3 foreign key modes now supported in Vitess - This enables seamless integration of VReplication with foreign keys. For more details on what operations Vitess takes please refer to the [design document for foreign keys](https://github.com/vitessio/vitess/issues/12967). 3. `FK_DISALLOW` - - In this mode Vitess explicitly disallows any DDL statements that try to create a foreign key constraint. This mode is equivalent to running Vtgates with the flag `--foreign_key_mode=disallow`. + In this mode Vitess explicitly disallows any DDL statements that try to create a foreign key constraint. This mode is equivalent to running VTGate with the flag `--foreign_key_mode=disallow`. #### Upgrade process -After upgrading from v17 to v18, the users should specify the correct foreign key mode for all their keyspaces in the Vschema using the new property. Once this change has taken effect, the deprecated flag `--foreign_key_mode` can be dropped from all the Vtgates. +After upgrading from v17 to v18, the users should specify the correct foreign key mode for all their keyspaces in the VSchema using the new property. +Once this change has taken effect, the deprecated flag `--foreign_key_mode` can be dropped from all the VTGates. ### VTAdmin #### vtadmin-web updated to node v18.16.0 (LTS) -Building vtadmin-web now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed +Building `vtadmin-web` now requires node >= v18.16.0 (LTS). Breaking changes from v16 to v18 are listed in https://nodejs.org/en/blog/release/v18.0.0, but none apply to VTAdmin. Full details on v18.16.0 are listed -here https://nodejs.org/en/blog/release/v18.16.0. +on https://nodejs.org/en/blog/release/v18.16.0. ### Deprecations and Deletions @@ -114,7 +113,6 @@ here https://nodejs.org/en/blog/release/v18.16.0. Throttler related `vttablet` flags: -- `--enable-lag-throttler` is now removed after being deprecated in `v17.0` - `--throttle_threshold` is deprecated and will be removed in `v19.0` - `--throttle_metrics_query` is deprecated and will be removed in `v19.0` - `--throttle_metrics_threshold` is deprecated and will be removed in `v19.0` @@ -135,11 +133,29 @@ Cache related `vtgate` flags: - `--gate_query_cache_lfu` is deprecated and will be removed in `v19.0`. The query cache always uses a LFU implementation now. - `--gate_query_cache_size` is deprecated and will be removed in `v19.0`. This option only applied to LRU caches, which are now unsupported. -VTGate flag: +VTGate flags: - `--schema_change_signal_user` is deprecated and will be removed in `v19.0` - `--foreign_key_mode` is deprecated and will be removed in `v19.0`. For more detail read the [foreign keys](#foreign-keys) section. +VDiff v1: + +[VDiff v2 was added in Vitess 15.0](https://vitess.io/blog/2022-11-22-vdiff-v2/) and marked as GA in 16.0. +The [legacy v1 client command](https://vitess.io/docs/18.0/reference/vreplication/vdiffv1/) is now deprecated in Vitess 18.0 and will be **removed** in 19.0. +Please switch all of your usage to the [new VDiff client](https://vitess.io/docs/18.0/reference/vreplication/vdiff/) command ASAP. + + +#### Deprecated Stats + +The following `EmergencyReparentShard` stats are deprecated in `v18.0` and will be removed in `v19.0`: +- `ers_counter` +- `ers_success_counter` +- `ers_failure_counter` + +These metrics are replaced by [new reparenting stats introduced in `v18.0`](#vtctld-and-vtorc-reparenting-stats). + +VTBackup stat `DurationByPhase` is deprecated. Use the binary-valued `Phase` stat instead. + #### Deleted Command Line Flags Flags in `vtcombo`: @@ -163,6 +179,7 @@ Flags in `vttablet`: - `--queryserver-config-stream-pool-prefill-parallelism` - `--queryserver-config-transaction-pool-prefill-parallelism` - `--queryserver-config-schema-change-signal-interval` +- `--enable-lag-throttler` Flags in `vtctld`: - `--vtctld_show_topology_crud` @@ -172,15 +189,6 @@ Flags in `vtorc`: - `--lock-shard-timeout` - `--orc_web_dir` -#### Deprecated Stats - -The following Emergency Reparent Shard stats are deprecated in `v18.0` and will be removed in `v19.0`: -- `ers_counter` -- `ers_success_counter` -- `ers_failure_counter` - -These metrics are replaced by [new reparenting stats introduced in `v18.0`](#vtctld-and-vtorc-reparenting-stats). - #### Deleted `v3` planner The `Gen4` planner has been the default planner since Vitess 14. The `v3` planner was deprecated in Vitess 15 and has now been removed in this release. @@ -198,14 +206,6 @@ The `vtgr` has been deprecated in Vitess 17, also see https://github.com/vitessi The undocumented `query_analyzer` binary has been removed in Vitess 18, see https://github.com/vitessio/vitess/issues/14054. -#### Deprecated VDiff v1 - -[VDiff v2 was added in Vitess 15.0](https://vitess.io/blog/2022-11-22-vdiff-v2/) and marked as GA in 16.0. The [legacy v1 client command](https://vitess.io/docs/18.0/reference/vreplication/vdiffv1/) is now deprecated in Vitess 18.0 and will be **removed** in 19.0. Please switch all of your usage to the [new VDiff client](https://vitess.io/docs/18.0/reference/vreplication/vdiff/) command ASAP. - -#### Deprecated VTbackup stat `DurationByPhase` - -VTBackup stat `DurationByPhase` is deprecated. Use the binary-valued `Phase` stat instead. - ### New stats #### VTGate Vindex unknown parameters @@ -218,10 +218,10 @@ In v17, the `vtbackup` stat `DurationByPhase` stat was added measuring the time `Phase` reports a 1 (active) or a 0 (inactive) for each of the following phases: - * `CatchupReplication` - * `InitialBackup` - * `RestoreLastBackup` - * `TakeNewBackup` +* `CatchupReplication` +* `InitialBackup` +* `RestoreLastBackup` +* `TakeNewBackup` To calculate how long `vtbackup` has spent in a given phase, sum the 1-valued data points over time and multiply by the data collection or reporting interval. For example, in Prometheus: @@ -232,9 +232,9 @@ sum_over_time(vtbackup_phase{phase="TakeNewBackup"}) * `PhaseStatus` reports a 1 (active) or a 0 (inactive) for each of the following phases and statuses: - * `CatchupReplication` phase has statuses `Stalled` and `Stopped`. - * `Stalled` is set to `1` when replication stops advancing. - * `Stopped` is set to `1` when replication stops before `vtbackup` catches up with the primary. +* `CatchupReplication` phase has statuses `Stalled` and `Stopped`. + * `Stalled` is set to `1` when replication stops advancing. + * `Stopped` is set to `1` when replication stops before `vtbackup` catches up with the primary. #### Backup and restore metrics for AWS S3 @@ -250,8 +250,8 @@ vtbackup_restore_count{component="BackupStorage",implementation="S3",operation=" #### VTCtld and VTOrc reparenting stats New VTCtld and VTorc stats were added to measure frequency of reparents by keyspace/shard: -- `emergency_reparent_counts` - Number of times Emergency Reparent Shard has been run. It is further subdivided by the keyspace, shard and the result of the operation. -- `planned_reparent_counts` - Number of times Planned Reparent Shard has been run. It is further subdivided by the keyspace, shard and the result of the operation. +- `emergency_reparent_counts` - Number of times `EmergencyReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. +- `planned_reparent_counts` - Number of times `PlannedReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. Also, the `reparent_shard_operation_timings` stat was added to provide per-operation timings of reparent operations. @@ -259,7 +259,7 @@ Also, the `reparent_shard_operation_timings` stat was added to provide per-opera #### New ResetSequences rpc -A new vttablet RPC `ResetSequences` has been added, which is being used by `MoveTables` and `Migrate` for workflows +A new VTTablet RPC `ResetSequences` has been added, which is being used by `MoveTables` and `Migrate` for workflows where a `sequence` table is being moved (https://github.com/vitessio/vitess/pull/13238). This has an impact on the Vitess upgrade process from an earlier version if you need to use such a workflow before the entire cluster is upgraded. @@ -283,7 +283,7 @@ Bullseye images will still be built and available as long as the OS build is cur #### Buster removed -Buster LTS supports will stop in June 2024, and Vitess v18.0 will be supported through October 2024. +Buster LTS supports will stop in June 2024, and Vitess v18.0 will be supported through October 2024. To prevent supporting a deprecated buster build for several months after June 2024, we are preemptively removing Vitess support. @@ -291,4 +291,4 @@ removing Vitess support. #### New Durability Policies -2 new inbuilt durability policies have been added to Vitess in this release namely `semi_sync_with_rdonly_ack` and `cross_cell_with_rdonly_ack`. These policies are exactly like `semi_sync` and `cross_cell` respectively, and differ just in the part where the rdonly tablets can also send semi-sync ACKs. +2 new inbuilt durability policies have been added to Vitess in this release namely `semi_sync_with_rdonly_ack` and `cross_cell_with_rdonly_ack`. These policies are exactly like `semi_sync` and `cross_cell` respectively, and differ just in the part where the rdonly tablets can also send semi-sync ACKs. \ No newline at end of file diff --git a/changelog/18.0/README.md b/changelog/18.0/README.md index 5b691530084..97676dc7e39 100644 --- a/changelog/18.0/README.md +++ b/changelog/18.0/README.md @@ -1,2 +1,4 @@ ## v18.0 * **[18.0.0](18.0.0)** + * [Changelog](18.0.0/changelog.md) + * [Release Notes](18.0.0/release_notes.md)