From e907c42ad6be90bb63972166ec4c7bd2630358d2 Mon Sep 17 00:00:00 2001 From: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Date: Thu, 13 Jun 2024 08:48:58 +0300 Subject: [PATCH] [main] Copy `v20.0.0-RC1` release notes (#16140) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> --- changelog/20.0/20.0.0/changelog.md | 530 +++++++++++++++++++++++++ changelog/20.0/20.0.0/release_notes.md | 369 +++++++++++++++++ changelog/20.0/README.md | 2 + 3 files changed, 901 insertions(+) create mode 100644 changelog/20.0/20.0.0/changelog.md create mode 100644 changelog/20.0/20.0.0/release_notes.md diff --git a/changelog/20.0/20.0.0/changelog.md b/changelog/20.0/20.0.0/changelog.md new file mode 100644 index 00000000000..4ee6cf6b298 --- /dev/null +++ b/changelog/20.0/20.0.0/changelog.md @@ -0,0 +1,530 @@ +# Changelog of Vitess v20.0.0 + +### Announcement +#### Governance + * add Tim Vaillancourt to maintainers [#15851](https://github.com/vitessio/vitess/pull/15851) +### Bug fixes +#### Authn/z + * Load `--grpc_auth_static_client_creds` file once [#15030](https://github.com/vitessio/vitess/pull/15030) +#### Backup and Restore + * Mysqld: capture mysqlbinlog std error output [#15278](https://github.com/vitessio/vitess/pull/15278) + * fix backup goroutine leak [#15410](https://github.com/vitessio/vitess/pull/15410) + * Ensure that WithParams keeps the transport [#15421](https://github.com/vitessio/vitess/pull/15421) + * Configurable incremental restore files path [#15451](https://github.com/vitessio/vitess/pull/15451) +#### Build/CI + * Use latest go version in update golang version workflow [#15159](https://github.com/vitessio/vitess/pull/15159) + * Fix `docker_build_images` CI workflow [#15635](https://github.com/vitessio/vitess/pull/15635) + * Continue building base/k8s when tag version is below v20 [#15638](https://github.com/vitessio/vitess/pull/15638) +#### CLI + * Fix some binaries to print the versions [#15306](https://github.com/vitessio/vitess/pull/15306) + * tablet: remove max-waiters setting [#15323](https://github.com/vitessio/vitess/pull/15323) +#### Cluster management + * Fix PromoteReplica call in ERS [#15934](https://github.com/vitessio/vitess/pull/15934) + * Remove the default for replica-net-timeout [#15938](https://github.com/vitessio/vitess/pull/15938) +#### Docker + * Add `mysqlbinlog` and `xtrabackup` to the `vitess/lite` image [#15775](https://github.com/vitessio/vitess/pull/15775) +#### Evalengine + * Fix type coercion between the sides of an UNION [#15340](https://github.com/vitessio/vitess/pull/15340) + * evalengine: Ensure to pass down the precision [#15611](https://github.com/vitessio/vitess/pull/15611) + * evalengine: Fix additional time type handling [#15614](https://github.com/vitessio/vitess/pull/15614) + * projection: Return correct collation information [#15801](https://github.com/vitessio/vitess/pull/15801) +#### General + * `ExecuteFetch`: error on multiple result sets [#14949](https://github.com/vitessio/vitess/pull/14949) + * GRPC: Address potential segfault in dedicated connection pooling [#15751](https://github.com/vitessio/vitess/pull/15751) + * Properly unescape keyspace name in FindAllShardsInKeyspace [#15765](https://github.com/vitessio/vitess/pull/15765) +#### Online DDL + * VReplication/OnlineDDL: reordering enum values [#15103](https://github.com/vitessio/vitess/pull/15103) + * Remove `ALGORITHM=COPY` from Online DDL in MySQL `>= 8.0.32` [#15376](https://github.com/vitessio/vitess/pull/15376) + * Enum value parsing: do not parse by whitespace [#15493](https://github.com/vitessio/vitess/pull/15493) + * Flaky test TestOnlineDDLVDiff: add additional check for vreplication workflow to exist [#15695](https://github.com/vitessio/vitess/pull/15695) + * `schemadiff`: `DROP COLUMN` not eligible for `INSTANT` algorithm if covered by an index [#15714](https://github.com/vitessio/vitess/pull/15714) + * `schemadiff` INSTANT DDL: impossible changes on tables with `FULLTEXT` index [#15725](https://github.com/vitessio/vitess/pull/15725) + * SchemaEngine: Ensure GetTableForPos returns table schema for "current" position by default [#15912](https://github.com/vitessio/vitess/pull/15912) +#### Query Serving + * Make connection killing resilient to MySQL hangs [#14500](https://github.com/vitessio/vitess/pull/14500) + * TxThrottler: dont throttle unless lag [#14789](https://github.com/vitessio/vitess/pull/14789) + * Planner Bug: Joins inside derived table [#14974](https://github.com/vitessio/vitess/pull/14974) + * fix: ignore internal tables in schema tracking [#15141](https://github.com/vitessio/vitess/pull/15141) + * bugfix: wrong field type returned for SUM [#15192](https://github.com/vitessio/vitess/pull/15192) + * Avoid rewriting unsharded queries and split semantic analysis in two [#15217](https://github.com/vitessio/vitess/pull/15217) + * Fix Delete with multi-tables related by foreign keys [#15218](https://github.com/vitessio/vitess/pull/15218) + * sqlparser: use integers instead of literals for Length/Precision [#15256](https://github.com/vitessio/vitess/pull/15256) + * Fix Go routine leaks in streaming calls [#15293](https://github.com/vitessio/vitess/pull/15293) + * Column alias expanding on ORDER BY [#15302](https://github.com/vitessio/vitess/pull/15302) + * planner: support union statements with ctes [#15312](https://github.com/vitessio/vitess/pull/15312) + * go/vt/discovery: use protobuf getters for SrvVschema [#15343](https://github.com/vitessio/vitess/pull/15343) + * Bugfix: GROUP BY/HAVING alias resolution [#15344](https://github.com/vitessio/vitess/pull/15344) + * SHOW VITESS_REPLICATION_STATUS: Only use replication tracker when it's enabled [#15348](https://github.com/vitessio/vitess/pull/15348) + * Fixing Column aliases in outer join queries [#15384](https://github.com/vitessio/vitess/pull/15384) + * Fix view tracking on sharded keyspace [#15436](https://github.com/vitessio/vitess/pull/15436) + * engine: fix race in concatenate [#15454](https://github.com/vitessio/vitess/pull/15454) + * Fix cycle detection for foreign keys [#15458](https://github.com/vitessio/vitess/pull/15458) + * fail insert when primary vindex cannot be mapped to a shard [#15500](https://github.com/vitessio/vitess/pull/15500) + * Fix aliasing in routes that have a derived table [#15550](https://github.com/vitessio/vitess/pull/15550) + * bugfix: handling of ANDed join predicates [#15551](https://github.com/vitessio/vitess/pull/15551) + * Make sure derived table column names are handled correctly [#15588](https://github.com/vitessio/vitess/pull/15588) + * Fix TPCH test by providing the correct field information in evalengine [#15623](https://github.com/vitessio/vitess/pull/15623) + * fix: don't forget DISTINCT for derived tables [#15672](https://github.com/vitessio/vitess/pull/15672) + * Prevent adding to query details after unserve common has started [#15684](https://github.com/vitessio/vitess/pull/15684) + * Fix panic in aggregation [#15728](https://github.com/vitessio/vitess/pull/15728) + * fix: use correct flag field for udf tracking [#15749](https://github.com/vitessio/vitess/pull/15749) + * Store Decimal precision and size while normalising [#15785](https://github.com/vitessio/vitess/pull/15785) + * Fix Scale and length handling in `CASE` and JOIN bind variables [#15787](https://github.com/vitessio/vitess/pull/15787) + * Fix derived table bug [#15831](https://github.com/vitessio/vitess/pull/15831) + * Fix CTE query by fixing bindvar pushing in unions [#15838](https://github.com/vitessio/vitess/pull/15838) + * Fix wrong assignment to `sql_id_opt` in the parser [#15862](https://github.com/vitessio/vitess/pull/15862) + * `schemadiff`: more `INSTANT` DDL compliance [#15871](https://github.com/vitessio/vitess/pull/15871) + * fix: handle info_schema routing [#15899](https://github.com/vitessio/vitess/pull/15899) + * fix: handle table_schema = '' without failing [#15901](https://github.com/vitessio/vitess/pull/15901) + * Fix aliasing in queries by keeping required projections [#15943](https://github.com/vitessio/vitess/pull/15943) + * connpool: Allow time out during shutdown [#15979](https://github.com/vitessio/vitess/pull/15979) + * `schemadiff`: assume default collation for textual column when collation is undefined [#16000](https://github.com/vitessio/vitess/pull/16000) + * fix: remove keyspace when merging subqueries [#16019](https://github.com/vitessio/vitess/pull/16019) +#### TabletManager + * mysqlctl: Improve handling of the lock file [#15404](https://github.com/vitessio/vitess/pull/15404) + * Fix possible race in MySQL startup and vttablet in parallel [#15538](https://github.com/vitessio/vitess/pull/15538) + * mysqlctld: setup a different default for onterm_timeout [#15575](https://github.com/vitessio/vitess/pull/15575) + * Fix the race condition during vttablet startup [#15731](https://github.com/vitessio/vitess/pull/15731) +#### Throttler + * Throttler: fix nil pointer dereference error [#15180](https://github.com/vitessio/vitess/pull/15180) + * Tablet throttler: starvation fix and consolidation of logic. [#15398](https://github.com/vitessio/vitess/pull/15398) + * Dedicated poolDialer logic for VTOrc, throttler [#15562](https://github.com/vitessio/vitess/pull/15562) + * Tablet throttler: recent check diff fix [#16001](https://github.com/vitessio/vitess/pull/16001) + * `ApplySchema`: reroute `ALTER VITESS_MIGRATION ... THROTTLE ...` via `UpdateThrottlerConfig` [#16030](https://github.com/vitessio/vitess/pull/16030) +#### Topology + * discovery: Fix tablets removed from healthcheck when topo server GetTablet call fails [#15633](https://github.com/vitessio/vitess/pull/15633) + * Fix ZooKeeper Topology connection locks not being cleaned up correctly [#15757](https://github.com/vitessio/vitess/pull/15757) +#### VReplication + * VReplication: disable foreign_key_checks for bulk data cleanup [#15261](https://github.com/vitessio/vitess/pull/15261) + * VReplication: Make Target Sequence Initialization More Robust [#15289](https://github.com/vitessio/vitess/pull/15289) + * MoveTables Atomic Mode: set FK checks off while deploying schema on targets [#15448](https://github.com/vitessio/vitess/pull/15448) + * VReplication: Fix workflow filtering in GetWorkflows RPC [#15524](https://github.com/vitessio/vitess/pull/15524) + * VReplication: Migrate intra-keyspace materialize workflows when Resharding the keyspace [#15536](https://github.com/vitessio/vitess/pull/15536) + * VReplication: Fix workflow update changed handling [#15621](https://github.com/vitessio/vitess/pull/15621) + * VReplication: Improve query buffering behavior during MoveTables traffic switching [#15701](https://github.com/vitessio/vitess/pull/15701) + * VReplication: Take replication lag into account in VStreamManager healthcheck result processing [#15761](https://github.com/vitessio/vitess/pull/15761) + * VReplication: Improve workflow cancel/delete [#15977](https://github.com/vitessio/vitess/pull/15977) + * [release-20.0-rc] vtctldclient: Apply (Shard | Keyspace| Table) Routing Rules commands don't work (#16096) [#16125](https://github.com/vitessio/vitess/pull/16125) + * [release-20.0] vtctldclient: Apply (Shard | Keyspace| Table) Routing Rules commands don't work (#16096) [#16126](https://github.com/vitessio/vitess/pull/16126) + * [release-20.0-rc] VReplication: Improve workflow cancel/delete (#15977) [#16130](https://github.com/vitessio/vitess/pull/16130) + * [release-20.0] VReplication: Improve workflow cancel/delete (#15977) [#16131](https://github.com/vitessio/vitess/pull/16131) +#### VTAdmin + * [VTAdmin API] Fix schema cache flag, add documentation [#15704](https://github.com/vitessio/vitess/pull/15704) + * [VTAdmin] Remove vtctld web link, improve local example (#15607) [#15824](https://github.com/vitessio/vitess/pull/15824) +#### VTCombo + * Correctly set log_dir default in vtcombo [#15153](https://github.com/vitessio/vitess/pull/15153) +#### VTorc + * Use the legacy name for timeouts [#15689](https://github.com/vitessio/vitess/pull/15689) + * Add timeout to all the contexts used for RPC calls in vtorc [#15991](https://github.com/vitessio/vitess/pull/15991) +#### vtexplain + * vtexplain: Fix setting up the column information [#15275](https://github.com/vitessio/vitess/pull/15275) + * vtexplain: Ensure memory topo is set up for throttler [#15279](https://github.com/vitessio/vitess/pull/15279) +#### vttestserver + * Revert unwanted logging change to `vttestserver` [#15148](https://github.com/vitessio/vitess/pull/15148) + * use proper mysql version in the `vttestserver` images [#15235](https://github.com/vitessio/vitess/pull/15235) +#### web UI + * [VTAdmin] Remove vtctld web link, improve local example [#15607](https://github.com/vitessio/vitess/pull/15607) +### CI/Build +#### Build/CI + * CI: Use v3 of fossa-action and exclude maven [#15140](https://github.com/vitessio/vitess/pull/15140) + * mysql: move colldump to its own standalone repository [#15166](https://github.com/vitessio/vitess/pull/15166) + * Remove concurrency group for check labels workflow [#15197](https://github.com/vitessio/vitess/pull/15197) + * CI: Use FOSSA push-only token for license scans on PRs [#15222](https://github.com/vitessio/vitess/pull/15222) + * Update FOSSA license scan links [#15233](https://github.com/vitessio/vitess/pull/15233) + * Update toolchain version in go.mod [#15245](https://github.com/vitessio/vitess/pull/15245) + * statsd: Update to datadog-go v5 API [#15247](https://github.com/vitessio/vitess/pull/15247) + * Ensure to use latest golangci-lint [#15413](https://github.com/vitessio/vitess/pull/15413) + * Fix go.mod [#15416](https://github.com/vitessio/vitess/pull/15416) + * bump `github.com/golang/protobuf` to `v1.5.4` [#15426](https://github.com/vitessio/vitess/pull/15426) + * Update all actions setup to latest versions [#15443](https://github.com/vitessio/vitess/pull/15443) + * CI: Disable CodeCov GitHub Changed Files Annotations [#15447](https://github.com/vitessio/vitess/pull/15447) + * Update to latest CodeQL [#15530](https://github.com/vitessio/vitess/pull/15530) + * Generate vtctldclient RPC client code from vtctlservice protobufs on make proto [#15561](https://github.com/vitessio/vitess/pull/15561) + * Add @mattlord as CODEOWNER for vtctld[client] related things [#15870](https://github.com/vitessio/vitess/pull/15870) + * Validate go versions in Static Code Checks CI [#15932](https://github.com/vitessio/vitess/pull/15932) + * Add CODEOWNERS for tablet throttler and schemadiff [#16036](https://github.com/vitessio/vitess/pull/16036) + * [release-20.0-rc] Add DCO workflow (#16052) [#16057](https://github.com/vitessio/vitess/pull/16057) + * [release-20.0] Add DCO workflow (#16052) [#16058](https://github.com/vitessio/vitess/pull/16058) + * [release-20.0-rc] Remove DCO workaround (#16087) [#16092](https://github.com/vitessio/vitess/pull/16092) + * [release-20.0] Remove DCO workaround (#16087) [#16093](https://github.com/vitessio/vitess/pull/16093) +#### General + * [main] Upgrade the Golang version to `go1.22.1` [#15405](https://github.com/vitessio/vitess/pull/15405) + * Upgrade go version to go1.22.2 [#15642](https://github.com/vitessio/vitess/pull/15642) + * [main] Upgrade the Golang version to `go1.22.3` [#15865](https://github.com/vitessio/vitess/pull/15865) + * [release-20.0] Upgrade the Golang version to `go1.22.4` [#16060](https://github.com/vitessio/vitess/pull/16060) + * [release-20.0-rc] [release-20.0] Upgrade the Golang version to `go1.22.4` (#16060) [#16064](https://github.com/vitessio/vitess/pull/16064) +#### Online DDL + * `onlineddl_scheduler` test: fix flakiness in artifact cleanup test [#15396](https://github.com/vitessio/vitess/pull/15396) + * Online DDL: fix flaky `onlineddl_scheduler` CI test [#16011](https://github.com/vitessio/vitess/pull/16011) +#### Throttler + * CI upgrade/downgrade tests for Online DDL / throttler / vreplication flow [#16017](https://github.com/vitessio/vitess/pull/16017) +#### VReplication + * VReplication: Get workflowFlavorVtctl endtoend testing working properly again [#15636](https://github.com/vitessio/vitess/pull/15636) +#### VTAdmin + * Update VTAdmin build script [#15839](https://github.com/vitessio/vitess/pull/15839) +### Dependencies +#### General + * Update to the latest x/net [#15680](https://github.com/vitessio/vitess/pull/15680) + * Revert GRPC context changes [#15780](https://github.com/vitessio/vitess/pull/15780) + * Upgrade the Golang Dependencies [#15823](https://github.com/vitessio/vitess/pull/15823) + * Upgrade the Golang Dependencies [#15942](https://github.com/vitessio/vitess/pull/15942) +#### Observability + * Bump vitess.io/vitess from 0.16.2 to 0.17.7 in /vitess-mixin [#15918](https://github.com/vitessio/vitess/pull/15918) + * Update `github.com/Azure/go-autorest/autorest/adal` to fix Dependabot alert [#15986](https://github.com/vitessio/vitess/pull/15986) +#### VTAdmin + * Bump vite from 4.5.2 to 4.5.3 in /web/vtadmin [#15634](https://github.com/vitessio/vitess/pull/15634) + * [release-20.0-rc] Update braces package (#16115) [#16119](https://github.com/vitessio/vitess/pull/16119) + * [release-20.0] Update braces package (#16115) [#16120](https://github.com/vitessio/vitess/pull/16120) +#### web UI + * Remove highcharts dependency pt. 1 [#15970](https://github.com/vitessio/vitess/pull/15970) +### Documentation +#### Authn/z + * Add v20 changelog docs for PR #15030 [#15367](https://github.com/vitessio/vitess/pull/15367) +#### Documentation + * Fix docs for unmanaged tablets [#15437](https://github.com/vitessio/vitess/pull/15437) + * [release-20.0-rc] Changelog 20.0: Fix broken links (#16048) [#16075](https://github.com/vitessio/vitess/pull/16075) + * [release-20.0] Changelog 20.0: Fix broken links (#16048) [#16076](https://github.com/vitessio/vitess/pull/16076) +#### General + * Add Shopify to `ADOPTERS.md` [#15853](https://github.com/vitessio/vitess/pull/15853) +#### Governance + * amend contributing guide to ban trivial contributions [#15618](https://github.com/vitessio/vitess/pull/15618) + * remove koz from active maintainer list [#15733](https://github.com/vitessio/vitess/pull/15733) +#### Topology + * Add lock shard docs [#15981](https://github.com/vitessio/vitess/pull/15981) +#### VReplication + * VDiff CLI: add missing target keyspace in VDiff command examples [#15525](https://github.com/vitessio/vitess/pull/15525) +#### VTGate + * Add changelogs for PR #15911 and #15919 [#16044](https://github.com/vitessio/vitess/pull/16044) +### Enhancement +#### Backup and Restore + * Point in time recovery and restore: assume (and validate) MySQL56 flavor in position arguments [#15599](https://github.com/vitessio/vitess/pull/15599) + * mysqlctl: Improve backup restore compatibility check [#15856](https://github.com/vitessio/vitess/pull/15856) +#### Build/CI + * [main] Add `release-19.0` to the auto go upgrade (#15157) [#15168](https://github.com/vitessio/vitess/pull/15168) + * Update paths filter action [#15254](https://github.com/vitessio/vitess/pull/15254) + * Add memory check for runners for VTOrc tests [#15317](https://github.com/vitessio/vitess/pull/15317) + * Assign and tag the release team for go update/upgrade auto PRs [#15737](https://github.com/vitessio/vitess/pull/15737) +#### Cluster management + * Add unmanaged tablet flag at vttablet level [#14871](https://github.com/vitessio/vitess/pull/14871) + * Fix error message for planned reparent shard [#15529](https://github.com/vitessio/vitess/pull/15529) + * Filter tablet map using valid candidates before reparenting to intermediate source [#15540](https://github.com/vitessio/vitess/pull/15540) + * Add a default for `replica_net_timeout` [#15663](https://github.com/vitessio/vitess/pull/15663) + * Add `GetServerStatus` RPC to use in PRS [#16022](https://github.com/vitessio/vitess/pull/16022) +#### Docker + * Remove MySQL/Percona from the `vitess/lite` Docker image [#15605](https://github.com/vitessio/vitess/pull/15605) + * Remove `vitess/base` and `vitess/k8s` Docker images [#15620](https://github.com/vitessio/vitess/pull/15620) +#### Driver + * Add types to Go SQL driver [#15569](https://github.com/vitessio/vitess/pull/15569) +#### Evalengine + * evalEngine: Implement SPACE, REVERSE [#15173](https://github.com/vitessio/vitess/pull/15173) + * evalengine: Implement LOCATE and friends [#15195](https://github.com/vitessio/vitess/pull/15195) + * evalEngine: Implement string `INSERT` [#15201](https://github.com/vitessio/vitess/pull/15201) + * evalengine: Implement BIN, OCT & CHAR functions [#15226](https://github.com/vitessio/vitess/pull/15226) + * evalEngine: Implement ELT and FIELD [#15249](https://github.com/vitessio/vitess/pull/15249) + * evalengine: Implement REPLACE [#15274](https://github.com/vitessio/vitess/pull/15274) + * evalengine: Implement `TO_SECONDS` [#15590](https://github.com/vitessio/vitess/pull/15590) + * evalengine: Fix temporal cases in `MAKETIME` [#15709](https://github.com/vitessio/vitess/pull/15709) + * evalengine: Implement `SEC_TO_TIME` [#15755](https://github.com/vitessio/vitess/pull/15755) + * evalengine: Add support for enum and set [#15783](https://github.com/vitessio/vitess/pull/15783) +#### Examples + * Update `operator.yaml` and add schedule backup example [#15969](https://github.com/vitessio/vitess/pull/15969) +#### General + * Enable gRPC Server Side Keepalive settings [#14939](https://github.com/vitessio/vitess/pull/14939) +#### Observability + * queryserving, observability: instrument vttablet query cache plan hits/misses [#14947](https://github.com/vitessio/vitess/pull/14947) + * VTGate Warnings: Add `WarnUnshardedOnly` to warnings counter [#15033](https://github.com/vitessio/vitess/pull/15033) + * VDiff: Add some stats [#15175](https://github.com/vitessio/vitess/pull/15175) +#### Online DDL + * DDL strategy flag `--unsafe-allow-foreign-keys` implies setting `FOREIGN_KEY_CHECKS=0` [#15432](https://github.com/vitessio/vitess/pull/15432) + * `schemadiff`: `SubsequentDiffStrategy`: allow/reject multiple changes on same entity [#15675](https://github.com/vitessio/vitess/pull/15675) + * Online DDL: unsupporting `gh-ost` DDL strategy [#15693](https://github.com/vitessio/vitess/pull/15693) + * Online DDL: better support for range partitioning [#15698](https://github.com/vitessio/vitess/pull/15698) +#### Query Serving + * Limit concurrent creation of healthcheck gRPC connections [#15053](https://github.com/vitessio/vitess/pull/15053) + * feat: use collation aware typing for UNION [#15122](https://github.com/vitessio/vitess/pull/15122) + * Fix evalEngine functions for dates on/before `0000-02-29` [#15124](https://github.com/vitessio/vitess/pull/15124) + * Subqueries in SET condition of UPDATE statement in presence of foreign keys [#15163](https://github.com/vitessio/vitess/pull/15163) + * Feature: Adding support for Vindex Hints to allow for greater control over shard routing [#15172](https://github.com/vitessio/vitess/pull/15172) + * Fix PRS from being blocked because of misbehaving clients [#15339](https://github.com/vitessio/vitess/pull/15339) + * Filter by keyspace earlier in `tabletgateway`s `WaitForTablets(...)` [#15347](https://github.com/vitessio/vitess/pull/15347) + * Update Planning for Limits in the presence of foreign keys [#15372](https://github.com/vitessio/vitess/pull/15372) + * `schemadiff`: supporting textual diff [#15388](https://github.com/vitessio/vitess/pull/15388) + * Use a throttled logger for exceeded memory warnings [#15424](https://github.com/vitessio/vitess/pull/15424) + * `schemadiff`: support valid foreign key cycles [#15431](https://github.com/vitessio/vitess/pull/15431) + * Handle panics during parallel execution [#15450](https://github.com/vitessio/vitess/pull/15450) + * Optimize with IN Clause for UPDATE/DELETE Statements on Vindexes [#15455](https://github.com/vitessio/vitess/pull/15455) + * `schemadiff`: remove `ForeignKeyLoopError` and loop detection logic [#15507](https://github.com/vitessio/vitess/pull/15507) + * Add support for `row_alias` syntax added in MySQL 8.0.19. [#15510](https://github.com/vitessio/vitess/pull/15510) + * test: failing unit test for type aggregation [#15518](https://github.com/vitessio/vitess/pull/15518) + * Allow non-reserved-keywords for index names [#15602](https://github.com/vitessio/vitess/pull/15602) + * feat: support IS UNKNOWN as synonym to IS NULL [#15673](https://github.com/vitessio/vitess/pull/15673) + * Use Kill Query for Non-Transaction Query Execution and Update Query Timeout / Cancelled Error Message [#15694](https://github.com/vitessio/vitess/pull/15694) + * Add schema tracking support for UDFs [#15705](https://github.com/vitessio/vitess/pull/15705) + * Gen4 Planner: support aggregate UDFs [#15710](https://github.com/vitessio/vitess/pull/15710) + * Prepare schema tracking for all UDFs [#15732](https://github.com/vitessio/vitess/pull/15732) + * add udfs to vschema on update [#15771](https://github.com/vitessio/vitess/pull/15771) + * fix: make sure string literals as columns are handled well [#15820](https://github.com/vitessio/vitess/pull/15820) + * Improve `mcmp` type comparison [#15821](https://github.com/vitessio/vitess/pull/15821) + * feat: optimise outer joins [#15840](https://github.com/vitessio/vitess/pull/15840) + * `schemadiff`: atomic diffs for range partition `DROP PARTITION` statement [#15843](https://github.com/vitessio/vitess/pull/15843) + * Add error transformer to vtgate executor [#15894](https://github.com/vitessio/vitess/pull/15894) + * allow query timeout hints on shard targeting [#15898](https://github.com/vitessio/vitess/pull/15898) + * feat: add support for WITH ROLLUP [#15930](https://github.com/vitessio/vitess/pull/15930) + * `schemadiff`: ALTER TABLE is not INSTANT-able if adding column with default expression value [#16028](https://github.com/vitessio/vitess/pull/16028) +#### TabletManager + * Introducing `ExecuteMultiFetchAsDba` gRPC and `vtctldclient ExecuteMultiFetchAsDBA` command [#15506](https://github.com/vitessio/vitess/pull/15506) +#### Throttler + * VReplication: Add throttler stats [#15221](https://github.com/vitessio/vitess/pull/15221) +#### Topology + * Topo: Add version support to GetTopologyPath [#15933](https://github.com/vitessio/vitess/pull/15933) +#### VReplication + * VReplication: Enforce consistent order for table copies and diffs [#15152](https://github.com/vitessio/vitess/pull/15152) + * VReplication: use proper column collations in vstreamer [#15313](https://github.com/vitessio/vitess/pull/15313) + * VStream: Allow for automatic resume after Reshard across VStreams [#15395](https://github.com/vitessio/vitess/pull/15395) + * VReplication: Remove auto_increment clauses for MoveTables to a sharded keyspace [#15679](https://github.com/vitessio/vitess/pull/15679) + * VReplication: Move ENUM and SET mappings from vplayer to vstreamer [#15723](https://github.com/vitessio/vitess/pull/15723) + * VReplication: Add stream DDL processing stats [#15769](https://github.com/vitessio/vitess/pull/15769) + * Improve WaitForPos errors, don't include Result struct in message [#15962](https://github.com/vitessio/vitess/pull/15962) +#### VTCombo + * [vtcombo] Expose `--tablet_types_to_wait` flag [#14951](https://github.com/vitessio/vitess/pull/14951) +#### VTGate + * Add sql text counts stats to `vtcombo`,`vtgate`+`vttablet` [#15897](https://github.com/vitessio/vitess/pull/15897) + * `vtgate`: support filtering tablets by tablet-tags [#15911](https://github.com/vitessio/vitess/pull/15911) + * Add support for sampling rate in `streamlog` [#15919](https://github.com/vitessio/vitess/pull/15919) +#### VTorc + * Improve VTOrc startup flow [#15315](https://github.com/vitessio/vitess/pull/15315) + * Add api end point to print the current database state in VTOrc [#15485](https://github.com/vitessio/vitess/pull/15485) + * Make `Durabler` interface methods public [#15548](https://github.com/vitessio/vitess/pull/15548) + * VTOrc: Rework recovery registration [#15591](https://github.com/vitessio/vitess/pull/15591) +#### vtctldclient + * `proto`: lexical ordering of `ExecuteMultiFetchAsDBA` [#15558](https://github.com/vitessio/vitess/pull/15558) +#### vttestserver + * Add initialize-with-vt-dba-tcp flag to enable TCP/IP connection access to the underlying MySQL instance [#15354](https://github.com/vitessio/vitess/pull/15354) +### Feature Request +#### Build/CI + * CI workflows: Split long running vreplication workflows [#15834](https://github.com/vitessio/vitess/pull/15834) +#### Cluster management + * [vtctldclient] Add GetShardReplication [#15389](https://github.com/vitessio/vitess/pull/15389) +#### Query Serving + * Update with Limit Plan [#15107](https://github.com/vitessio/vitess/pull/15107) + * Add support for Update Multi Table [#15211](https://github.com/vitessio/vitess/pull/15211) + * Delete with subquery support [#15219](https://github.com/vitessio/vitess/pull/15219) + * Multi Target Delete Support [#15294](https://github.com/vitessio/vitess/pull/15294) + * Feature: Multi Target Update Support [#15402](https://github.com/vitessio/vitess/pull/15402) + * Foreign Key: Add support for multi target delete [#15504](https://github.com/vitessio/vitess/pull/15504) + * Foreign Key: Add support for Multi Table and Multi Target Update Statement [#15523](https://github.com/vitessio/vitess/pull/15523) + * Respect Straight Join in Vitess query planning [#15528](https://github.com/vitessio/vitess/pull/15528) + * feat: Add support for Insert with row alias [#15790](https://github.com/vitessio/vitess/pull/15790) + * Add support for multi table update with non literal value [#15980](https://github.com/vitessio/vitess/pull/15980) +#### Throttler + * Tablet throttler: adding more stats [#15224](https://github.com/vitessio/vitess/pull/15224) +#### VReplication + * Experimental: Multi-tenant import support in Vitess [#15503](https://github.com/vitessio/vitess/pull/15503) + * VDiff/OnlineDDL: add support for running VDiffs for OnlineDDL migrations [#15546](https://github.com/vitessio/vitess/pull/15546) + * Multi-tenant MoveTables: Create vreplication streams only on specified shards [#15746](https://github.com/vitessio/vitess/pull/15746) + * Multi-tenant MoveTables: allow switching replica/rdonly traffic separately before switching primary traffic [#15768](https://github.com/vitessio/vitess/pull/15768) + * Multi-tenant migrations: add topo locking while updating keyspace routing rules [#15807](https://github.com/vitessio/vitess/pull/15807) +#### VTorc + * VTOrc optimize TMC usage [#15356](https://github.com/vitessio/vitess/pull/15356) + * VTOrc checks and fixes replication misconfiguration issues [#15881](https://github.com/vitessio/vitess/pull/15881) +### Internal Cleanup +#### Backup and Restore + * endtoend: Remove usage of deprecated terminology [#15827](https://github.com/vitessio/vitess/pull/15827) +#### Build/CI + * [e2e] More vtctldclient updates in tests [#15276](https://github.com/vitessio/vitess/pull/15276) + * wranger: Clean up leak check and use existing version [#15334](https://github.com/vitessio/vitess/pull/15334) + * update andrew's email [#15495](https://github.com/vitessio/vitess/pull/15495) + * Remove self-hosted runners in ci_workflow_gen [#15989](https://github.com/vitessio/vitess/pull/15989) + * Linkname removal (step 1) [#16016](https://github.com/vitessio/vitess/pull/16016) +#### Cluster management + * go/vt/wrangler: pass reparent options structs [#15251](https://github.com/vitessio/vitess/pull/15251) + * delete TestActionAndTimeout [#15322](https://github.com/vitessio/vitess/pull/15322) + * Deprecate old reparent metrics and replace with new ones [#16031](https://github.com/vitessio/vitess/pull/16031) +#### Docker + * Revert the removal of the MySQL binaries in the `vitess/lite` image [#16042](https://github.com/vitessio/vitess/pull/16042) +#### Examples + * Update env.sh so that is does not error when running on Mac [#15835](https://github.com/vitessio/vitess/pull/15835) + * Local Examples: Add --binary-as-hex=false flag to mysql alias [#15996](https://github.com/vitessio/vitess/pull/15996) +#### General + * New for loops and some assert/require [#15194](https://github.com/vitessio/vitess/pull/15194) + * Remove loopclosure captures from tests [#15202](https://github.com/vitessio/vitess/pull/15202) + * Make `--pprof-http` default to false [#15260](https://github.com/vitessio/vitess/pull/15260) + * discovery: Remove unused code [#15332](https://github.com/vitessio/vitess/pull/15332) + * chore: remove repetitive words [#15449](https://github.com/vitessio/vitess/pull/15449) + * Migrate to math/rand/v2 [#15513](https://github.com/vitessio/vitess/pull/15513) + * Fix misorganized annotations [#15566](https://github.com/vitessio/vitess/pull/15566) + * changelogs: squash 19.0.2/19.0.3 into just 19.0.3 and remove 19.0.2 [#15665](https://github.com/vitessio/vitess/pull/15665) + * Cleanup usage of FLUSH PRIVILEGES [#15700](https://github.com/vitessio/vitess/pull/15700) + * Upgrade the Golang Dependencies [#15743](https://github.com/vitessio/vitess/pull/15743) + * grpc: Always pass through context for dialer [#15781](https://github.com/vitessio/vitess/pull/15781) + * Switch to use semisync source / replica plugins [#15791](https://github.com/vitessio/vitess/pull/15791) + * Use replica queries when available [#15808](https://github.com/vitessio/vitess/pull/15808) +#### Messaging + * messager: add consistent log prefix w/ table name [#15973](https://github.com/vitessio/vitess/pull/15973) +#### Observability + * Upgrade the golang version used `vitess-mixin` [#15972](https://github.com/vitessio/vitess/pull/15972) +#### Online DDL + * New unified internal table names format: part 2, generating new names [#15178](https://github.com/vitessio/vitess/pull/15178) +#### Query Serving + * Make tablet collation mismatch warning throttled [#15123](https://github.com/vitessio/vitess/pull/15123) + * schemadiff: Clean up MySQL version from diff hints [#15210](https://github.com/vitessio/vitess/pull/15210) + * refactor: change FuncExpr to use Exprs instead of SelectExprs [#15368](https://github.com/vitessio/vitess/pull/15368) + * refactor: clean up semantics package [#15385](https://github.com/vitessio/vitess/pull/15385) + * planbuilder: Cleanup unused logic [#15415](https://github.com/vitessio/vitess/pull/15415) + * cleanup: make sure we use the right Offset [#15576](https://github.com/vitessio/vitess/pull/15576) + * Add more fields to the marshal output of column [#15622](https://github.com/vitessio/vitess/pull/15622) + * modify error message when transaction not found in numbered pool [#15760](https://github.com/vitessio/vitess/pull/15760) + * Delete the deprecated pool size flags [#15844](https://github.com/vitessio/vitess/pull/15844) + * refactor: introduce helper method to extract logic [#15939](https://github.com/vitessio/vitess/pull/15939) + * refactor: remove logical plan interface [#16006](https://github.com/vitessio/vitess/pull/16006) + * Decouple topotools from vschema [#16008](https://github.com/vitessio/vitess/pull/16008) +#### TabletManager + * srvtopo: Setup metrics in init() function [#15304](https://github.com/vitessio/vitess/pull/15304) + * Revert "Skip for-loop alloc in `go/vt/discovery/healthcheck.go`" [#15328](https://github.com/vitessio/vitess/pull/15328) + * mysqlctld: Remove unneeded resets in init_db.sql [#15832](https://github.com/vitessio/vitess/pull/15832) + * mysql: Handle more deprecated SQL commands [#15907](https://github.com/vitessio/vitess/pull/15907) +#### Throttler + * Throttler: refactor stats variables [#15574](https://github.com/vitessio/vitess/pull/15574) + * Tablet throttler: remove `LowPriority` logic [#16013](https://github.com/vitessio/vitess/pull/16013) +#### Topology + * topo: Clean up unused code [#15515](https://github.com/vitessio/vitess/pull/15515) + * Etcd2Topo: Use node's ModRevision consistently for in-memory topo.Version value [#15847](https://github.com/vitessio/vitess/pull/15847) + * Fix documentation for `--lock-timeout` [#16021](https://github.com/vitessio/vitess/pull/16021) +#### VReplication + * Remove Usage of VReplicationExec For _vt.vreplication Reads [#14424](https://github.com/vitessio/vitess/pull/14424) + * VReplication: improve reliability of log management [#15374](https://github.com/vitessio/vitess/pull/15374) + * delete unused code in vreplication e2e tests [#15378](https://github.com/vitessio/vitess/pull/15378) + * MoveTables: remove option to specify source keyspace alias for multi-tenant migrations [#15712](https://github.com/vitessio/vitess/pull/15712) + * Delete the deprecated vreplication tablet type flag [#15857](https://github.com/vitessio/vitess/pull/15857) + * VReplication: Remove noisy logs [#15987](https://github.com/vitessio/vitess/pull/15987) + * VReplication: refactor denied tables unit test, add couple more tests [#15995](https://github.com/vitessio/vitess/pull/15995) +#### VTAdmin + * Update Node version to current LTS release [#15822](https://github.com/vitessio/vitess/pull/15822) +#### VTTablet + * go/cmd: Audit and fix context.Background() usage [#15928](https://github.com/vitessio/vitess/pull/15928) +#### VTorc + * Remove unneeded loading of the MySQL driver [#15502](https://github.com/vitessio/vitess/pull/15502) + * vtorc: Cleanup unused code [#15508](https://github.com/vitessio/vitess/pull/15508) + * Remove reading emergently instances [#15580](https://github.com/vitessio/vitess/pull/15580) + * Cleanup unused vtorc code [#15595](https://github.com/vitessio/vitess/pull/15595) + * VTOrc: Cleanup node registration and unused code [#15617](https://github.com/vitessio/vitess/pull/15617) + * Remove unused code in VTOrc [#15813](https://github.com/vitessio/vitess/pull/15813) + * vtorc: Switch to Vitess stats [#15948](https://github.com/vitessio/vitess/pull/15948) + * Deprecate old metrics in VTOrc and replace with new ones [#15994](https://github.com/vitessio/vitess/pull/15994) +#### vtctl + * Remove legacy `EmergencyReparentShard` stats [#15941](https://github.com/vitessio/vitess/pull/15941) +### Other +#### Other + * Match MySQL's `LAST_INSERT_ID` behaviour [#15697](https://github.com/vitessio/vitess/pull/15697) +### Performance +#### General + * prevent vtctld from creating tons of S3 connections [#15296](https://github.com/vitessio/vitess/pull/15296) +#### Query Serving + * Skip for-loop alloc in `go/vt/discovery/healthcheck.go` [#15326](https://github.com/vitessio/vitess/pull/15326) + * logstats: do not allocate memory while logging [#15539](https://github.com/vitessio/vitess/pull/15539) + * rewrite shuffleTablets to be clearer and more efficient [#15716](https://github.com/vitessio/vitess/pull/15716) +#### TabletManager + * Fix: transition to `math/rand/v2` for Improved Performance and Code Clarity [#15438](https://github.com/vitessio/vitess/pull/15438) +#### VReplication + * VReplication Workflows (RowStreamer): explicitly set read only when creating snapshots in the copy phase [#15690](https://github.com/vitessio/vitess/pull/15690) +#### VTTablet + * Improve performance for `BaseShowTablesWithSizes` query. [#15713](https://github.com/vitessio/vitess/pull/15713) + * Do not load table stats when booting `vttablet`. [#15715](https://github.com/vitessio/vitess/pull/15715) + * [release-20.0] Do not load table stats when booting `vttablet`. (#15715) [#16101](https://github.com/vitessio/vitess/pull/16101) +### Regression +#### Query Serving + * Fix routing rule query rewrite [#15253](https://github.com/vitessio/vitess/pull/15253) + * fix: remove keyspace from column during query builder [#15514](https://github.com/vitessio/vitess/pull/15514) + * Fix regression where inserts into reference tables with a different name on sharded keyspaces were not routed correctly. [#15796](https://github.com/vitessio/vitess/pull/15796) + * fix: derived table join column expression to be part of add join predicate on rewrite [#15956](https://github.com/vitessio/vitess/pull/15956) + * fix: insert on duplicate update to add list argument in the bind variables map [#15961](https://github.com/vitessio/vitess/pull/15961) + * [release-20.0-rc] fix: order by subquery planning (#16049) [#16133](https://github.com/vitessio/vitess/pull/16133) + * [release-20.0] fix: order by subquery planning (#16049) [#16134](https://github.com/vitessio/vitess/pull/16134) +#### Throttler + * Enable 'heartbeat_on_demand_duration' in local/examples [#15204](https://github.com/vitessio/vitess/pull/15204) +#### vttestserver + * Fix logging issue when running in Docker with the syslog daemon disabled [#15176](https://github.com/vitessio/vitess/pull/15176) +### Release +#### General + * Bump to `v20.0.0-SNAPSHOT` after the `v19.0.0-RC1` release [#15138](https://github.com/vitessio/vitess/pull/15138) + * Copy `v19.0.0-RC1` release notes on `main` [#15164](https://github.com/vitessio/vitess/pull/15164) + * Copy `v19.0.0` release notes on `main` [#15417](https://github.com/vitessio/vitess/pull/15417) + * Copy `v17.0.6` release notes on `main` [#15486](https://github.com/vitessio/vitess/pull/15486) + * Copy `v18.0.3` release notes on `main` [#15488](https://github.com/vitessio/vitess/pull/15488) + * Copy `v19.0.1` release notes on `main` [#15490](https://github.com/vitessio/vitess/pull/15490) + * Copy `v19.0.2` release notes on `main` [#15647](https://github.com/vitessio/vitess/pull/15647) + * Copy `v18.0.4` release notes on `main` [#15659](https://github.com/vitessio/vitess/pull/15659) + * Copy `v19.0.3` release notes on `main` [#15661](https://github.com/vitessio/vitess/pull/15661) + * Copy `v18.0.5` release notes on `main` [#15886](https://github.com/vitessio/vitess/pull/15886) + * Copy `v19.0.4` release notes on `main` [#15887](https://github.com/vitessio/vitess/pull/15887) + * Copy `v17.0.7` release notes on `main` [#15890](https://github.com/vitessio/vitess/pull/15890) + * [release-20.0-rc] Code Freeze for `v20.0.0-RC1` [#16046](https://github.com/vitessio/vitess/pull/16046) + * Bump to `v21.0.0-SNAPSHOT` after the `v20.0.0-RC1` release [#16047](https://github.com/vitessio/vitess/pull/16047) +### Testing +#### Build/CI + * Rewrite _many_ tests to use vtctldclient invocations, mostly non-output related stuff [#15270](https://github.com/vitessio/vitess/pull/15270) + * [e2e] vtctld init tablet and some output-based commands [#15297](https://github.com/vitessio/vitess/pull/15297) + * CI: Address data races on memorytopo Conn.closed [#15365](https://github.com/vitessio/vitess/pull/15365) + * Reduce excessing logging in CI [#15462](https://github.com/vitessio/vitess/pull/15462) + * Split unit test and unit race into 2 components [#15734](https://github.com/vitessio/vitess/pull/15734) + * CI: Address data race in TestSchemaVersioning [#15998](https://github.com/vitessio/vitess/pull/15998) +#### CLI + * Add required tests for `internal/flag` [#15220](https://github.com/vitessio/vitess/pull/15220) + * test: Add missing tests and refactor existing tests for `go/flagutil` [#15789](https://github.com/vitessio/vitess/pull/15789) +#### Cluster management + * Fix Data race in tests introduced in #15934 [#15993](https://github.com/vitessio/vitess/pull/15993) +#### General + * Added unit tests for cmd/internal/docgen package [#15019](https://github.com/vitessio/vitess/pull/15019) + * unit test for go/yaml2/yaml.go [#15027](https://github.com/vitessio/vitess/pull/15027) + * Added unit tests for `go/cmd/rulesctl/` package [#15028](https://github.com/vitessio/vitess/pull/15028) + * Added tests for the go/trace package [#15052](https://github.com/vitessio/vitess/pull/15052) + * Added missing tests for the go/streamlog package [#15064](https://github.com/vitessio/vitess/pull/15064) + * Added unit tests for vt/grpcclient package [#15072](https://github.com/vitessio/vitess/pull/15072) + * modernize various tests [#15184](https://github.com/vitessio/vitess/pull/15184) + * go1.22: remove outdated loopclosure captures in tests [#15227](https://github.com/vitessio/vitess/pull/15227) + * chore: modernize tests [#15244](https://github.com/vitessio/vitess/pull/15244) + * Add required tests for `go/netutil` [#15392](https://github.com/vitessio/vitess/pull/15392) + * Add required tests for `go/stats/opentsdb` [#15394](https://github.com/vitessio/vitess/pull/15394) + * test: Replace `t.fatalf` with testify `require` in `go/vt/schemamanager` [#15600](https://github.com/vitessio/vitess/pull/15600) + * Remove mysql 5.7 tests that are no longer required [#15809](https://github.com/vitessio/vitess/pull/15809) + * Fix unit-test-runner bug [#15815](https://github.com/vitessio/vitess/pull/15815) + * test: Add tests for `go/ioutil` and refactor existing [#15885](https://github.com/vitessio/vitess/pull/15885) + * test: Add required tests for `vt/key`, `timer` and `cache/theine/bf` [#15976](https://github.com/vitessio/vitess/pull/15976) +#### Observability + * VStreamer: add throttled logs when row/result/vstreamers get throttled. [#14936](https://github.com/vitessio/vitess/pull/14936) +#### Query Serving + * test: skip should mark the correct *testing.T [#15333](https://github.com/vitessio/vitess/pull/15333) + * test: Add required tests for `go/mysql/collations/charset` [#15435](https://github.com/vitessio/vitess/pull/15435) + * test: Add missing tests for `go/mysql/datetime` [#15501](https://github.com/vitessio/vitess/pull/15501) + * schemadiff: add `EntityDiffByStatement`, a testing friendly utility [#15519](https://github.com/vitessio/vitess/pull/15519) + * `schemadiff`: better nil check validation [#15526](https://github.com/vitessio/vitess/pull/15526) + * Add testing for shard scoped foreign keys [#15571](https://github.com/vitessio/vitess/pull/15571) + * chore: use interface instead of struct for tests [#15581](https://github.com/vitessio/vitess/pull/15581) + * Fix AVG() sharded planning [#15626](https://github.com/vitessio/vitess/pull/15626) + * Run launchable in unit race [#15686](https://github.com/vitessio/vitess/pull/15686) + * Make upgrade downgrade tests faster by removing redundancy [#15687](https://github.com/vitessio/vitess/pull/15687) + * Fix Foreign key fuzzer to ignore rows affected [#15841](https://github.com/vitessio/vitess/pull/15841) + * `schemadiff`: adding charset/collation tests [#15872](https://github.com/vitessio/vitess/pull/15872) + * test: Add required tests for `go/logstats` [#15893](https://github.com/vitessio/vitess/pull/15893) + * test: Add missing/required tests for `sqltypes` and `mathstats` [#15920](https://github.com/vitessio/vitess/pull/15920) + * test: Cleaner plan tests output [#15922](https://github.com/vitessio/vitess/pull/15922) +#### TabletManager + * test: Add missing tests for `go/vt/mysqlctl` [#15585](https://github.com/vitessio/vitess/pull/15585) + * test: Add e2e tests for `replication` [#15671](https://github.com/vitessio/vitess/pull/15671) +#### Throttler + * test: Use testify require/assert instead of t.Fatal/Error in `go/vt/throttler` [#15703](https://github.com/vitessio/vitess/pull/15703) + * v20 backport: CI upgrade/downgrade tests for Online DDL / throttler / vreplication flow [#16065](https://github.com/vitessio/vitess/pull/16065) + * [release-20.0-rc] v20 backport: CI upgrade/downgrade tests for Online DDL / throttler / vreplication flow (#16065) [#16082](https://github.com/vitessio/vitess/pull/16082) +#### VReplication + * VStreamer Unit Tests: framework to remove the need to specify serialized strings in row events for unit tests [#14903](https://github.com/vitessio/vitess/pull/14903) + * VtctldClient Reshard: add e2e tests to confirm CLI options and fix discovered issues. [#15353](https://github.com/vitessio/vitess/pull/15353) + * VStreamer unit test: port remaining tests to new framework [#15366](https://github.com/vitessio/vitess/pull/15366) + * VReplication: Fix vtctldclient SwitchReads related bugs and move the TestBasicV2Workflows e2e test to vtctldclient [#15579](https://github.com/vitessio/vitess/pull/15579) + * VStreamer unit tests: refactor pending test [#15845](https://github.com/vitessio/vitess/pull/15845) +#### VTorc + * Add missing tests for `go/vt/vtorc/collection` [#15070](https://github.com/vitessio/vitess/pull/15070) +#### vtctldclient + * Add tests for GetTablets partial results [#15829](https://github.com/vitessio/vitess/pull/15829) +#### vttestserver + * Add `no_scatter` flag to vttestserver [#15670](https://github.com/vitessio/vitess/pull/15670) + diff --git a/changelog/20.0/20.0.0/release_notes.md b/changelog/20.0/20.0.0/release_notes.md new file mode 100644 index 00000000000..ffcf8057a15 --- /dev/null +++ b/changelog/20.0/20.0.0/release_notes.md @@ -0,0 +1,369 @@ +# Release of Vitess v20.0.0 + +## Summary + +### Table of Contents + +- **[Major Changes](#major-changes)** + - **[Deletions](#deletions)** + - [`--vreplication_tablet_type` flag](#vreplication-tablet-type-deletion) + - [Pool Capacity Flags](#pool-flags-deletion) + - [vitess/base and vitess/k8s Docker images](#base-k8s-images) + - [`gh-ost` binary and endtoend tests](#gh-ost-binary-tests-removal) + - [Legacy `EmergencyReparentShard` stats](#legacy-emergencyshardreparent-stats) + - **[Breaking changes](#breaking-changes)** + - [Metric Name Changes in VTOrc](#metric-change-vtorc) + - [ENUM and SET column handling in VTGate VStream API](#enum-set-vstream) + - [`shutdown_grace_period` Default Change](#shutdown-grace-period-default) + - [New `unmanaged` Flag and `disable_active_reparents` deprecation](#unmanaged-flag) + - [`recovery-period-block-duration` Flag deprecation](#recovery-block-deprecation) + - [`mysqlctld` `onterm-timeout` Default Change](#mysqlctld-onterm-timeout) + - [`MoveTables` now removes `auto_increment` clauses by default when moving tables from an unsharded keyspace to a sharded one](#move-tables-auto-increment) + - [`Durabler` interface method renaming](#durabler-interface-method-renaming) + - **[Query Compatibility](#query-compatibility)** + - [Vindex Hints](#vindex-hints) + - [Update with Limit Support](#update-limit) + - [Update with Multi Table Support](#multi-table-update) + - [Update with Multi Target Support](#update-multi-target) + - [Delete with Subquery Support](#delete-subquery) + - [Delete with Multi Target Support](#delete-multi-target) + - [User Defined Functions Support](#udf-support) + - [Insert Row Alias Support](#insert-row-alias-support) + - **[Query Timeout](#query-timeout)** + - **[Flag changes](#flag-changes)** + - [`pprof-http` default change](#pprof-http-default) + - [New `healthcheck-dial-concurrency` flag](#healthcheck-dial-concurrency-flag) + - [New minimum for `--buffer_min_time_between_failovers`](#buffer_min_time_between_failovers-flag) + - [New `track-udfs` vtgate flag](#vtgate-track-udfs-flag) + - [Help text fix for `--lock-timeout`](#documentation-lock-timeout) + - [New `--querylog-sample-rate` flag](#querylog-sample-rate-flag) + - [New `--tablet-filter-tags` flag](#tablet-filter-tags-flag) + +- **[Minor Changes](#minor-changes)** + - **[New Stats](#new-stats)** + - [VTTablet Query Cache Hits and Misses](#vttablet-query-cache-hits-and-misses) + - [VTGate and VTTablet Query Text Characters Processed](#vttablet-query-text-characters-processed) + - **[`SIGHUP` reload of gRPC client static auth creds](#sighup-reload-of-grpc-client-auth-creds)** + - **[VTAdmin](#vtadmin)** + - [Updated to node v20.12.2](#updated-node) + - [Replaced highcharts with d3](#replaced-highcharts) + +## Major Changes + +### Deletion + +#### `--vreplication_tablet_type` flag + +The previously deprecated flag `--vreplication_tablet_type` has been deleted. + +#### Pool Capacity Flags + +The previously deprecated flags `--queryserver-config-query-pool-waiter-cap`, `--queryserver-config-stream-pool-waiter-cap` and `--queryserver-config-txpool-waiter-cap` have been deleted. + +#### `vitess/base` and `vitess/k8s` Docker images + +Since we have deleted MySQL from our `vitess/lite` image, we are removing the `vitess/base` and `vitess/k8s` images. + +These images are no longer useful since we can use `vitess/lite` as the base of many other Docker images (`vitess/vtgate`, `vitess/vtgate`, ...). + +#### `gh-ost` binary and endtoend tests + +Vitess 20.0 drops support for `gh-ost` DDL strategy. + +`vttablet` binary no longer embeds a `gh-ost` binary. Users of `gh-ost` DDL strategy will need to supply a `gh-ost` binary on the `vttablet` host or pod. Vitess will look for the `gh-ost` binary in the system `PATH`; otherwise the user should supply `vttablet --gh-ost-path`. + +Vitess' endtoend tests no longer use nor test `gh-ost` migrations. + +#### Legacy `EmergencyReparentShard` stats + +The following `EmergencyReparentShard` stats were deprecated in Vitess 18.0 and are removed in Vitess 20.0: +- `ers_counter` +- `ers_success_counter` +- `ers_failure_counter` + +These counters are replaced by the following stats _(introduced in Vitess 18.0)_: +- `emergency_reparent_counts` - Number of times `EmergencyReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. +- `planned_reparent_counts` - Number of times `PlannedReparentShard` has been run. It is further subdivided by the keyspace, shard and the result of the operation. + +Also, the `reparent_shard_operation_timings` stat was added to provide per-operation timings of reparent operations. + +### Breaking Changes + +#### Metric Name Changes in VTOrc + +The following metric names have been changed in VTOrc. The old metrics are still available in `/debug/vars` for this release, but will be removed in later releases. The new metric names and the deprecated metric names resolve to the same metric name on prometheus, so there is no change there. + +| Old Metric Name | New Metric Name | Name in Prometheus | +|:--------------------------------------------:|:----------------------------------------:|:--------------------------------------------------:| +| `analysis.change.write` | `AnalysisChangeWrite` | `vtorc_analysis_change_write` | +| `audit.write` | `AuditWrite` | `vtorc_audit_write` | +| `discoveries.attempt` | `DiscoveriesAttempt` | `vtorc_discoveries_attempt` | +| `discoveries.fail` | `DiscoveriesFail` | `vtorc_discoveries_fail` | +| `discoveries.instance_poll_seconds_exceeded` | `DiscoveriesInstancePollSecondsExceeded` | `vtorc_discoveries_instance_poll_seconds_exceeded` | +| `discoveries.queue_length` | `DiscoveriesQueueLength` | `vtorc_discoveries_queue_length` | +| `discoveries.recent_count` | `DiscoveriesRecentCount` | `vtorc_discoveries_recent_count` | +| `instance.read` | `InstanceRead` | `vtorc_instance_read` | +| `instance.read_topology` | `InstanceReadTopology` | `vtorc_instance_read_topology` | +| `emergency_reparent_counts` | `EmergencyReparentCounts` | `vtorc_emergency_reparent_counts` | +| `planned_reparent_counts` | `PlannedReparentCounts` | `vtorc_planned_reparent_counts` | +| `reparent_shard_operation_timings` | `ReparentShardOperationTimings` | `vtorc_reparent_shard_operation_timings_bucket` | + + + +#### ENUM and SET column handling in VTGate VStream API + +The [VTGate VStream API](https://vitess.io/docs/reference/vreplication/vstream/) now returns [`ENUM`](https://dev.mysql.com/doc/refman/en/enum.html) and [`SET`](https://dev.mysql.com/doc/refman/en/set.html) column type values in [`VEvent`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent) messages (in the embedded [`RowChange`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#RowChange) messages) as their string values instead of the integer based ones — in both the copy/snapshot phase and the streaming phase. This change was done to make the `VStream` API more user-friendly, intuitive, and to align the behavior across both phases. Before [this change](https://github.com/vitessio/vitess/pull/15723) the values for [`ENUM`](https://dev.mysql.com/doc/refman/en/enum.html) and [`SET`](https://dev.mysql.com/doc/refman/en/set.html) columns were string values in the copy phase but integer values (which only have an internal meaning to MySQL) in the streaming phase. This inconsistency led to various [challenges and issues](https://github.com/vitessio/vitess/issues/15750) for each `VStream` client/consumer (e.g. the [`Debezium` Vitess connector](https://debezium.io/documentation/reference/stable/connectors/vitess.html) failed to properly perform a snapshot for tables containing these column types). Now the behavior is intuitive — clients need the string values as the eventual sink is often not MySQL so each consumer needed to perform the mappings themselves — and consistent. While this is a (potentially) breaking change, a new boolean field has been added to the [`FieldEvent`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#FieldEvent) message called `EnumSetStringValues`. When that field is `false` (in Vitess v19 and older) then the consumer will need to perform the mappings during streaming phase, but not during copy phase. When this field is `true`, then no mapping is required. This will help to ensure a smooth transition for all consumers over time. To demonstrate, let's look at the textual output (printing the received `VEvents` as strings) when streaming a single `enum_set_test` table from the unsharded `commerce` keyspace so that we can see what the VStream looks like before and after when we start a new VStream in copy/snapshot mode and then transition to streaming mode for the following table: + +```sql +CREATE TABLE `enum_set_test` ( + `id` int NOT NULL AUTO_INCREMENT, + `name` varchar(120) DEFAULT NULL, + `shirt_size` enum('small','medium','large','xlarge','xxlarge') DEFAULT NULL, + `hobbies` set('knitting','cooking','pickleball','biking','hiking','motorcycle','video games','reading') DEFAULT NULL, + PRIMARY KEY (`id`) +) +``` + +And with the table having this data when we start our `VStream` and begin the copy/snapshot phase: + +```sql +mysql> select * from enum_set_test; ++----+-----------+------------+-------------------------+ +| id | name | shirt_size | hobbies | ++----+-----------+------------+-------------------------+ +| 1 | Billy Bob | xlarge | cooking,reading | +| 2 | Sally Mae | medium | knitting,cooking,hiking | ++----+-----------+------------+-------------------------+ +2 rows in set (0.00 sec) +``` + +And finally we will perform the following inserts and updates to the table during the streaming phase: + +```sql +insert into enum_set_test values (3, "Matt Lord", 'medium', 'pickleball,biking,hiking,motorcycle,video games,reading'); +insert into enum_set_test values (4, "Jerry Badyellow", 'large', ''); +update enum_set_test set shirt_size = 'small', hobbies = 'knitting,cooking,hiking,reading' where id = 2; +``` + +Vitess v19 and older: + +```text +[type:BEGIN keyspace:"commerce" shard:"0" type:FIELD field_event:{table_name:"commerce.enum_set_test" fields:{name:"id" type:INT32 table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"id" column_length:11 charset:63 flags:49667 column_type:"int"} fields:{name:"name" type:VARCHAR table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"name" column_length:480 charset:255 column_type:"varchar(120)"} fields:{name:"shirt_size" type:ENUM table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"shirt_size" column_length:28 charset:255 flags:256 column_type:"enum('small','medium','large','xlarge','xxlarge')"} fields:{name:"hobbies" type:SET table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"hobbies" column_length:288 charset:255 flags:2048 column_type:"set('knitting','cooking','pickleball','biking','hiking','motorcycle','video games','reading')"} keyspace:"commerce" shard:"0"} keyspace:"commerce" shard:"0"] +[type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-35"}} keyspace:"commerce" shard:"0"] +[type:ROW row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:6 lengths:15 values:"1Billy Bobxlargecooking,reading"}} keyspace:"commerce" shard:"0"} keyspace:"commerce" shard:"0" type:ROW row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:6 lengths:23 values:"2Sally Maemediumknitting,cooking,hiking"}} keyspace:"commerce" shard:"0"} keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-35" table_p_ks:{table_name:"enum_set_test" lastpk:{fields:{name:"id" type:INT32 charset:63 flags:49667} rows:{lengths:1 values:"2"}}}}} keyspace:"commerce" shard:"0" type:COMMIT keyspace:"commerce" shard:"0"] +[type:BEGIN keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-35"}} keyspace:"commerce" shard:"0" type:COMMIT keyspace:"commerce" shard:"0"] +[type:COPY_COMPLETED keyspace:"commerce" shard:"0" type:COPY_COMPLETED] +[type:BEGIN timestamp:1715179728 current_time:1715179728532658000 keyspace:"commerce" shard:"0" type:FIELD timestamp:1715179728 field_event:{table_name:"commerce.enum_set_test" fields:{name:"id" type:INT32 table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"id" column_length:11 charset:63 flags:49667 column_type:"int"} fields:{name:"name" type:VARCHAR table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"name" column_length:480 charset:255 column_type:"varchar(120)"} fields:{name:"shirt_size" type:ENUM table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"shirt_size" column_length:28 charset:255 flags:256 column_type:"enum('small','medium','large','xlarge','xxlarge')"} fields:{name:"hobbies" type:SET table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"hobbies" column_length:288 charset:255 flags:2048 column_type:"set('knitting','cooking','pickleball','biking','hiking','motorcycle','video games','reading')"} keyspace:"commerce" shard:"0"} current_time:1715179728535652000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179728 row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:1 lengths:3 values:"3Matt Lord2252"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179728535739000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-36"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179728 current_time:1715179728535754000 keyspace:"commerce" shard:"0"] +[type:BEGIN timestamp:1715179735 current_time:1715179735538607000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179735 row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:15 lengths:1 lengths:1 values:"4Jerry Badyellow30"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179735538659000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-37"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179735 current_time:1715179735538672000 keyspace:"commerce" shard:"0"] +[type:BEGIN timestamp:1715179741 current_time:1715179741728690000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179741 row_event:{table_name:"commerce.enum_set_test" row_changes:{before:{lengths:1 lengths:9 lengths:1 lengths:2 values:"2Sally Mae219"} after:{lengths:1 lengths:9 lengths:1 lengths:3 values:"2Sally Mae1147"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179741728730000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/ce357206-0d49-11ef-8fd1-a74564279579:1-38"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179741 current_time:1715179741728744000 keyspace:"commerce" shard:"0"] +``` + +Vitess v20 and newer: + +```text +[type:BEGIN keyspace:"commerce" shard:"0" type:FIELD field_event:{table_name:"commerce.enum_set_test" fields:{name:"id" type:INT32 table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"id" column_length:11 charset:63 flags:49667 column_type:"int"} fields:{name:"name" type:VARCHAR table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"name" column_length:480 charset:255 column_type:"varchar(120)"} fields:{name:"shirt_size" type:ENUM table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"shirt_size" column_length:28 charset:255 flags:256 column_type:"enum('small','medium','large','xlarge','xxlarge')"} fields:{name:"hobbies" type:SET table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"hobbies" column_length:288 charset:255 flags:2048 column_type:"set('knitting','cooking','pickleball','biking','hiking','motorcycle','video games','reading')"} keyspace:"commerce" shard:"0" enum_set_string_values:true} keyspace:"commerce" shard:"0"] +[type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-50"}} keyspace:"commerce" shard:"0"] +[type:ROW row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:6 lengths:15 values:"1Billy Bobxlargecooking,reading"}} keyspace:"commerce" shard:"0"} keyspace:"commerce" shard:"0" type:ROW row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:6 lengths:23 values:"2Sally Maemediumknitting,cooking,hiking"}} keyspace:"commerce" shard:"0"} keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-50" table_p_ks:{table_name:"enum_set_test" lastpk:{fields:{name:"id" type:INT32 charset:63 flags:49667} rows:{lengths:1 values:"2"}}}}} keyspace:"commerce" shard:"0" type:COMMIT keyspace:"commerce" shard:"0"] +[type:BEGIN keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-50"}} keyspace:"commerce" shard:"0" type:COMMIT keyspace:"commerce" shard:"0"] +[type:COPY_COMPLETED keyspace:"commerce" shard:"0" type:COPY_COMPLETED] +[type:BEGIN timestamp:1715179399 current_time:1715179399817221000 keyspace:"commerce" shard:"0" type:FIELD timestamp:1715179399 field_event:{table_name:"commerce.enum_set_test" fields:{name:"id" type:INT32 table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"id" column_length:11 charset:63 flags:49667 column_type:"int"} fields:{name:"name" type:VARCHAR table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"name" column_length:480 charset:255 column_type:"varchar(120)"} fields:{name:"shirt_size" type:ENUM table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"shirt_size" column_length:28 charset:255 flags:256 column_type:"enum('small','medium','large','xlarge','xxlarge')"} fields:{name:"hobbies" type:SET table:"enum_set_test" org_table:"enum_set_test" database:"vt_commerce" org_name:"hobbies" column_length:288 charset:255 flags:2048 column_type:"set('knitting','cooking','pickleball','biking','hiking','motorcycle','video games','reading')"} keyspace:"commerce" shard:"0" enum_set_string_values:true} current_time:1715179399821735000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179399 row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:9 lengths:6 lengths:55 values:"3Matt Lordmediumpickleball,biking,hiking,motorcycle,video games,reading"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179399821762000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-51"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179399 current_time:1715179399821801000 keyspace:"commerce" shard:"0"] +[type:BEGIN timestamp:1715179399 current_time:1715179399822310000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179399 row_event:{table_name:"commerce.enum_set_test" row_changes:{after:{lengths:1 lengths:15 lengths:5 lengths:0 values:"4Jerry Badyellowlarge"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179399822355000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-52"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179399 current_time:1715179399822360000 keyspace:"commerce" shard:"0"] +[type:BEGIN timestamp:1715179400 current_time:1715179400512056000 keyspace:"commerce" shard:"0" type:ROW timestamp:1715179400 row_event:{table_name:"commerce.enum_set_test" row_changes:{before:{lengths:1 lengths:9 lengths:6 lengths:23 values:"2Sally Maemediumknitting,cooking,hiking"} after:{lengths:1 lengths:9 lengths:5 lengths:31 values:"2Sally Maesmallknitting,cooking,hiking,reading"}} keyspace:"commerce" shard:"0" flags:1} current_time:1715179400512094000 keyspace:"commerce" shard:"0" type:VGTID vgtid:{shard_gtids:{keyspace:"commerce" shard:"0" gtid:"MySQL56/156f702a-0d47-11ef-8723-653d045ab990:1-53"}} keyspace:"commerce" shard:"0" type:COMMIT timestamp:1715179400 current_time:1715179400512108000 keyspace:"commerce" shard:"0"] +``` + +An example key difference there being that `after:{lengths:1 lengths:9 lengths:1 lengths:3 values:"2Sally Mae1147"}` from Vitess v19 and older becomes `after:{lengths:1 lengths:9 lengths:5 lengths:31 values:"2Sally Maesmallknitting,cooking,hiking,reading"}` from Vitess v20 and newer. So `1` -> `small` and `147` -> `knitting,cooking,hiking,reading` for the `ENUM` and `SET` column values respectively. This also demonstrates why this mapping is necessary in consumers/clients, as `147` has no logical meaning/value for this column outside of MySQL internals. + +If you're using the [`Debezium` Vitess connector](https://debezium.io/documentation/reference/stable/connectors/vitess.html), you should upgrade your connector to 2.7 (the next release) — which should contain [the relevant necessary changes](https://issues.redhat.com/browse/DBZ-7792) — *prior to upgrading Vitess* to v20.0.1 or later. If you're using any of the PlanetScale connectors ([`AirByte`](https://github.com/planetscale/airbyte-source/), [`FiveTran`](https://github.com/planetscale/fivetran-source), or [`singer-tap`](https://github.com/planetscale/singer-tap)) then no actions are required. + +If you're using a custom `VStream` client/consumer, then you will need to build a new client with the updated v20 [binlogdata protos](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata) ([source](https://github.com/vitessio/vitess/blob/main/proto/binlogdata.proto) for which would be in `main` or the `release-20.0` branch) before needing to support Vitess v20.0.1 or later. Your client will then be able to handle old and new messages, with older messages always having this new field set to `false`. + +#### `shutdown_grace_period` Default Change + +The `--shutdown_grace_period` flag, which was introduced in v2 with a default of `0 seconds`, has now been changed to default to `3 seconds`. +This makes reparenting in Vitess resilient to client errors, and prevents PlannedReparentShard from timing out. + +In order to preserve the old behaviour, the users can set the flag back to `0 seconds` causing open transactions to never be shutdown, but in that case, they run the risk of PlannedReparentShard calls timing out. + +#### New `unmanaged` Flag and `disable_active_reparents` deprecation + +New flag `--unmanaged` has been introduced in this release to make it easier to flag unmanaged tablets. It also runs validations to make sure the unmanaged tablets are configured properly. `--disable_active_reparents` flag has been deprecated for `vttablet`, `vtcombo` and `vttestserver` binaries and will be removed in future releases. Specifying the `--unmanaged` flag will also block replication commands and replication repairs. + +Starting this release, all unmanaged tablets should specify this flag. + + +#### `recovery-period-block-duration` Flag deprecation + +The flag `--recovery-period-block-duration` has been deprecated in VTOrc from this release. Its value is now ignored and the flag will be removed in later releases. +VTOrc no longer blocks recoveries for a certain duration after a previous recovery has completed. Since VTOrc refreshes the required information after +acquiring a shard lock, blocking of recoveries is not required. + +#### `mysqlctld` `onterm_timeout` Default Change + +The `--onterm_timeout` flag default value has changed for `mysqlctld`. It now is by default long enough to be able to wait for the default `--shutdown-wait-time` when shutting down on a `TERM` signal. + +This is necessary since otherwise MySQL would never shut down cleanly with the old defaults, since `mysqlctld` would shut down already after 10 seconds by default. + +#### `MoveTables` now removes `auto_increment` clauses by default when moving tables from an unsharded keyspace to a sharded one + +A new `--remove-sharded-auto-increment` flag has been added to the [`MoveTables` create sub-command](https://vitess.io/docs/20.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/) and it is set to `true` by default. This flag controls whether any [MySQL `auto_increment`](https://dev.mysql.com/doc/refman/en/example-auto-increment.html) clauses should be removed from the table definitions when moving tables from an unsharded keyspace to a sharded one. This is now done by default as `auto_increment` clauses should not typically be used with sharded tables and you should instead rely on externally generated values such as a form of universally/globally unique identifiers or use [Vitess sequences](https://vitess.io/docs/reference/features/vitess-sequences/) in order to ensure that each row has a unique identifier (Primary Key value) across all shards. If for some reason you want to retain them you can set this new flag to `false` when creating the workflow. + +#### `Durabler` interface method renaming + +The methods of [the `Durabler` interface](https://github.com/vitessio/vitess/blob/main/go/vt/vtctl/reparentutil/durability.go#L70-L79) in `go/vt/vtctl/reparentutil` were renamed to be public _(capitalized)_ methods to make it easier to integrate custom Durability Policies from external packages. See [RFC for details](https://github.com/vitessio/vitess/issues/15544). + +Users of custom Durability Policies must rename private `Durabler` methods. + +Changes: +- The `promotionRule` method was renamed to `PromotionRule` +- The `semiSyncAckers` method was renamed to `SemiSyncAckers` +- The `isReplicaSemiSync` method was renamed to `IsReplicaSemiSync` + +### Query Compatibility + +#### Vindex Hints + +Vitess now supports Vindex hints that provide a way for users to influence the shard routing of queries in Vitess by specifying, which vindexes should be considered or ignored by the query planner. This feature enhances the control over query execution, allowing for potentially more efficient data access patterns in sharded databases. + +Example: + ```sql + SELECT * FROM user USE VINDEX (hash_user_id, secondary_vindex) WHERE user_id = 123; + SELECT * FROM order IGNORE VINDEX (range_order_id) WHERE order_date = '2021-01-01'; + ``` + +For more information about Vindex hints and its usage, please consult the documentation. + +#### Update with Limit Support + +Support is added for sharded update with limit. + +Example: `update t1 set t1.foo = 'abc', t1.bar = 23 where t1.baz > 5 limit 1` + +More details about how it works is available in [MySQL Docs](https://dev.mysql.com/doc/refman/8.0/en/update.html) + +#### Update with Multi Table Support + +Support is added for sharded multi-table update with column update on single target table using multiple table join. + +Example: `update t1 join t2 on t1.id = t2.id join t3 on t1.col = t3.col set t1.baz = 'abc', t1.apa = 23 where t3.foo = 5 and t2.bar = 7` + +More details about how it works is available in [MySQL Docs](https://dev.mysql.com/doc/refman/8.0/en/update.html) + +#### Update with Multi Target Support + +Support is added for sharded multi table target update. + +Example: `update t1 join t2 on t1.id = t2.id set t1.foo = 'abc', t2.bar = 23` + +More details about how it works is available in [MySQL Docs](https://dev.mysql.com/doc/refman/8.0/en/update.html) + +#### Delete with Subquery Support + +Support is added for sharded table delete with subquery + +Example: `delete from t1 where id in (select col from t2 where foo = 32 and bar = 43)` + +#### Delete with Multi Target Support + +Support is added for sharded multi table target delete. + +Example: `delete t1, t3 from t1 join t2 on t1.id = t2.id join t3 on t1.col = t3.col` + +More details about how it works is available in [MySQL Docs](https://dev.mysql.com/doc/refman/8.0/en/delete.html) + +#### User Defined Functions Support + +VTGate can track any user defined functions for better planning. +User Defined Functions (UDFs) should be directly loaded in the underlying MySQL. + +It should be enabled in VTGate with the `--track-udfs` flag. +This will enable the tracking of UDFs in VTGate and will be used for planning. +Without this flag, VTGate will not be aware that there might be aggregating user-defined functions in the query that need to be pushed down to MySQL. + +More details about how to load UDFs is available in [MySQL Docs](https://dev.mysql.com/doc/extending-mysql/8.0/en/adding-loadable-function.html) + +#### Insert Row Alias Support + +Support is added to have row alias in Insert statement to be used with `on duplicate key update`. + +Example: +- `insert into user(id, name, email) valies (100, 'Alice', 'alice@mail.com') as new on duplicate key update name = new.name, email = new.email` +- `insert into user(id, name, email) valies (100, 'Alice', 'alice@mail.com') as new(m, n, p) on duplicate key update name = n, email = p` + +More details about how it works is available in [MySQL Docs](https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html) + +### Query Timeout +On a query timeout, Vitess closed the connection using the `kill connection` statement. This leads to connection churn +which is not desirable in some cases. To avoid this, Vitess now uses the `kill query` statement to cancel the query. +This will only cancel the query and does not terminate the connection. + +### Flag Changes + +#### `pprof-http` Default Change + +The `--pprof-http` flag, which was introduced in v19 with a default of `true`, has now been changed to default to `false`. +This makes HTTP `pprof` endpoints now an *opt-in* feature, rather than opt-out. +To continue enabling these endpoints, explicitly set `--pprof-http` when starting up Vitess components. + +#### New `--healthcheck-dial-concurrency` flag + +The new `--healthcheck-dial-concurrency` flag defines the maximum number of healthcheck connections that can open concurrently. This limit is to avoid hitting Go runtime panics on deployments watching enough tablets [to hit the runtime's maximum thread limit of `10000`](https://pkg.go.dev/runtime/debug#SetMaxThreads) due to blocking network syscalls. This flag applies to `vtcombo`, `vtctld` and `vtgate` only and a value less than the runtime max thread limit _(`10000`)_ is recommended. + +#### New minimum for `--buffer_min_time_between_failovers` + +The `--buffer_min_time_between_failovers` `vttablet` flag now has a minimum value of `1s`. This is because a value of 0 can cause issues with the buffering mechanics resulting in unexpected and unnecessary query errors — in particular during `MoveTables SwitchTraffic` operations. If you are currently specifying a value of 0 for this flag then you will need to update the config value to 1s *prior to upgrading to v20 or later* as `vttablet` will report an error and terminate if you attempt to start it with a value of 0. + +#### New `--track-udfs` vtgate flag + +The new `--track-udfs` flag enables VTGate to track user defined functions for better planning. + +#### Help text fix for `--lock-timeout` + +The help text for the flag `--lock-timeout` was incorrect. We were documenting it as a flag that controlled the duration for which the shard lock was acquired. It is actually the maximum duration for which we wait while attempting to acquire a lock from the topology server. + +#### New `--querylog-sample-rate` flag + +The new flag `--querylog-sample-rate float` adds support for sampling queries based on a float value between 0.0 _(no logging)_ and 1.0 _(all queries logged)_. If configured, this filtering is applied after the existing `--querylog-filter-tag` filter. + +#### New `--tablet-filter-tags` flag + +The new flag `--tablet-filter-tags StringMap` adds support to VTGate for filtering tablets by tablet tag key/values, specified as comma-separated list of key:values. The tags of a tablet are defined by the VTTablet flag `--init_tags`, which is also defined as a comma-separated list of key:values. + +## Minor Changes + +### New Stats + +#### VTTablet Query Cache Hits and Misses + +VTTablet exposes two new counter stats: + + * `QueryCacheHits`: Query engine query cache hits + * `QueryCacheMisses`: Query engine query cache misses + +### VTTablet Query Text Characters Processed + +VTGate and VTTablet expose a new counter stat `QueryTextCharactersProcessed` to reflect the number of query text characters processed. + +VTGate groups this metric by Operation, Keyspace and TabletType. On VTTablet it is grouped by Table, Plan and optionally Workload. + +### `SIGHUP` reload of gRPC client static auth creds + +The internal gRPC client now caches the static auth credentials and supports reloading via the `SIGHUP` signal. Previous to v20 the credentials were not cached. They were re-loaded from disk on every use. + +### VTAdmin + +#### vtadmin-web updated to node v20.12.2 (LTS) + +Building `vtadmin-web` now requires node >= v20.12.0 (LTS). Breaking changes from v18 to v20 can be found at https://nodejs.org/en/blog/release/v20.12.0 -- with no known issues that apply to VTAdmin. +Full details on the node v20.12.2 release can be found at https://nodejs.org/en/blog/release/v20.12.2. + +#### Replaced highcharts with d3 + +The vtadmin-web UI no longer has a dependency on highcharts for licensing reasons. The tablet QPS, tablet VReplication QPS, and workflow streams lag charts have all been replaced by d3. We'll be iteratively improving the d3 charts until they reach feature parity with the original highcharts charts. + +------------ +The entire changelog for this release can be found [here](https://github.com/vitessio/vitess/blob/main/changelog/20.0/20.0.0/changelog.md). + +The release includes 410 merged Pull Requests. + +Thanks to all our contributors: @Aoang, @GuptaManan100, @Its-Maniaco, @Maniktherana, @VaibhavMalik4187, @ajm188, @aparajon, @app/dependabot, @app/github-actions, @app/vitess-bot, @arthurschreiber, @bddicken, @beingnoble03, @brendar, @crazeteam, @dbussink, @deepthi, @demmer, @derekperkins, @ejortegau, @frouioui, @harshit-gangal, @mattlord, @maxenglander, @mdlayher, @notfelineit, @pavedroad, @rafer, @rohit-nayak-ps, @rvrangel, @shlomi-noach, @systay, @timvaillancourt, @tycol7, @vitess-bot, @vmg, @wangweicugw, @whuang8, @yoheimuta + diff --git a/changelog/20.0/README.md b/changelog/20.0/README.md index 4fb70ae78c1..a143396b60c 100644 --- a/changelog/20.0/README.md +++ b/changelog/20.0/README.md @@ -1,2 +1,4 @@ ## v20.0 * **[20.0.0](20.0.0)** + * [Changelog](20.0.0/changelog.md) + * [Release Notes](20.0.0/release_notes.md)