Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VStream: Add notes about reshard handling #1700

Merged
merged 1 commit into from
Mar 19, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 35 additions & 26 deletions content/en/docs/20.0/reference/vreplication/vstream.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,20 @@ that allows clients to subscribe to a change event stream for a set of tables.

## Use Cases

* **Change Data Capture (CDC)**: `VStream` can be used to capture changes to a
table and send them to a downstream system. This is useful for building
real-time data pipelines.
* **Change Data Capture (CDC)**: `VStream` can be used to capture changes to a
table and send them to a downstream system. This is useful for building
real-time data pipelines.

## Overview

`VStream` supports copying the current contents of a table — as you will often not
have the binary logs going back to the creation of the table — and then begin streaming
new changes to the table from that point on. It also supports resuming this initial copy
phase if it's interrupted for any reason.
`VStream` supports copying the current contents of a table — as you will often not have the binary logs going back
to the creation of the table — and then begin streaming new changes to the table from that point on. It supports
resuming this initial copy phase if it's interrupted for any reason. It also supports automatic handling of
[resharding events](../reshard/) — if the `VStream` is connected throughout then it will automatically transition from
the old shards to the new when traffic is switched ([`SwitchTraffic` or `ReverseTraffic`](../reshard/#switchtraffic)), and
if you were not connected but re-connect after traffic is switched ([`SwitchTraffic` or `ReverseTraffic`](../reshard/#switchtraffic))
*but before the old shards are removed*, it will automatically catch up on any missed changes on the old shards before
seamlessly transitioning to the new shards and continuing to stream all changes made there.

Events in the stream are [MySQL row based binary log events](https://dev.mysql.com/doc/refman/en/mysqlbinlog-row-events.html) — with [extended metadata](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent)
— and can be processed by event bridges which support Vitess such as
Expand Down Expand Up @@ -150,25 +154,28 @@ a [`VStreamReader`](https://pkg.go.dev/vitess.io/vitess/go/vt/vtgate/vtgateconn#
the stream could not be initialized. You would call the `Recv` method on that
[`VStreamReader`](https://pkg.go.dev/vitess.io/vitess/go/vt/vtgate/vtgateconn#VStreamReader) in a for loop and
responses will be sent when available. Each response consisting of the following two parameters:
* An array of [`VEvent`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent) objects — the new messages to process in the stream
* An `error` — an error that, if non-nil, indicates the stream has been closed (`EOF`) or an error occurred

* An array of [`VEvent`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent) objects — the new messages to process in the stream
* An `error` — an error that, if non-nil, indicates the stream has been closed (`EOF`) or an error occurred

### API Types
* [TabletType](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/topodata#TabletType)
* [VGtid](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VGtid)
* [ShardGtid](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#ShardGtid)
* [Filter.Rule](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#Rule)
* [LastPKEvent](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#LastPKEvent)
* [TableLastPK](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#TableLastPK)
* [VStreamFlags](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/vtgate#VStreamFlags)
* [VStreamReader](https://pkg.go.dev/vitess.io/vitess/go/vt/vtgate/vtgateconn#VStreamReader)
* [VEvent](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent)

* [TabletType](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/topodata#TabletType)
* [VGtid](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VGtid)
* [ShardGtid](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#ShardGtid)
* [Filter.Rule](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#Rule)
* [LastPKEvent](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#LastPKEvent)
* [TableLastPK](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#TableLastPK)
* [VStreamFlags](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/vtgate#VStreamFlags)
* [VStreamReader](https://pkg.go.dev/vitess.io/vitess/go/vt/vtgate/vtgateconn#VStreamReader)
* [VEvent](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/binlogdata#VEvent)

### Example Usage

You can find a full example go client [here](https://github.com/vitessio/vitess/blob/main/examples/local/vstream_client.go).

Below is a snippet showing how to use the `VStream` API in go:

```go
gconn, err := vtgateconn.Dial(ctx, grpcAddress)
if err != nil {
Expand Down Expand Up @@ -269,6 +276,7 @@ Therefore, please exercise caution when using regular expressions in production.
## Debugging

There is also an SQL interface that can be used for testing and debugging from a `vtgate`. Here's an example:

```mysql
$ mysql --quick <vtgate params>

Expand Down Expand Up @@ -297,15 +305,16 @@ customer_id: 1
```

## Monitoring

VTGates publish vstream metrics listed [here](../metrics/#vtgate-metrics).

## More Reading

* [VStream Copy](https://github.com/vitessio/vitess/issues/6277)
* [VStream API and Resharding](../internal/vstream-stream-migration/)
* [VStream Skew Minimization](../internal/vstream-skew-detection/)
* Debezium Connector for Vitess
* [Docs](https://debezium.io/documentation/reference/stable/connectors/vitess.html)
* [Source](https://github.com/debezium/debezium-connector-vitess/)
* Blog posts
* [Streaming Vitess at Bolt](https://medium.com/bolt-labs/streaming-vitess-at-bolt-f8ea93211c3f)
* [VStream Copy](https://github.com/vitessio/vitess/issues/6277)
* [VStream API and Resharding](../internal/vstream-stream-migration/)
* [VStream Skew Minimization](../internal/vstream-skew-detection/)
* Debezium Connector for Vitess
* [Docs](https://debezium.io/documentation/reference/stable/connectors/vitess.html)
* [Source](https://github.com/debezium/debezium-connector-vitess/)
* Blog posts
* [Streaming Vitess at Bolt](https://medium.com/bolt-labs/streaming-vitess-at-bolt-f8ea93211c3f)