Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VReplication: Support binlog_row_value_options=PARTIAL_JSON #17345

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3c337a4
WiP
mattlord Dec 8, 2024
c9cf71c
Remove `set names binary` in controller
mattlord Dec 9, 2024
2aed1aa
Fix test
mattlord Dec 9, 2024
7b75ed6
Use charset literals in partial JSON expression
mattlord Dec 9, 2024
748589d
Minor changes
mattlord Dec 9, 2024
0b2b9cf
Remove DEBUG printfs
mattlord Dec 9, 2024
7be99a3
Get N diffs working
mattlord Dec 9, 2024
bd3aa67
Build out unit test
mattlord Dec 9, 2024
96ffca4
Enable PARTIAL_JSON for vrepl e2e tests
mattlord Dec 9, 2024
69afa52
Merge remote-tracking branch 'origin/main' into vrepl_partial_json
mattlord Dec 9, 2024
42d5db4
Get rid of unused func
mattlord Dec 10, 2024
6e288df
Get binlog_row_image = noblob and binlog-row-value-options = PARTIAL_…
mattlord Dec 10, 2024
f2a5445
Get edge cases working
mattlord Dec 10, 2024
1e492b9
Minor changes from self review
mattlord Dec 11, 2024
4fbbe49
Add new unit test cases
mattlord Dec 11, 2024
4d15ca2
Fix bug with NULL JSON columns
mattlord Dec 11, 2024
d6f132a
Merge remote-tracking branch 'origin/main' into vrepl_partial_json
mattlord Dec 11, 2024
8171e14
Add test queries to e2e test
mattlord Dec 11, 2024
dd324ef
Properly handle PK changes
mattlord Dec 12, 2024
eb3f1b0
Add unit test
mattlord Dec 13, 2024
91b1e33
Support optional unit test behavior for 5.7
mattlord Dec 13, 2024
1980b89
Try upgrading codecov to see if that helps
mattlord Dec 14, 2024
7b25395
Minor changes from self review
mattlord Dec 14, 2024
799850d
Improve comments
mattlord Dec 15, 2024
e68c035
Minor readability improvement
mattlord Dec 15, 2024
8ad1fc0
Nitting while waiting for reviews
mattlord Dec 19, 2024
c168ff0
Add another unit test case
mattlord Dec 19, 2024
dcec5f6
Update unit test name and comment to reflect content
mattlord Dec 19, 2024
64e0422
Another self review
mattlord Dec 21, 2024
a230cb1
Error nits (send halp)
mattlord Dec 21, 2024
f23f73c
Grow byte buffer to exact size
mattlord Dec 21, 2024
76cd316
Merge remote-tracking branch 'origin/main' into vrepl_partial_json
mattlord Dec 21, 2024
93aa37c
Add relese note summary section
mattlord Dec 21, 2024
6264397
Add more unit test cases
mattlord Dec 22, 2024
3a172c0
Simplify buffer allocation
mattlord Dec 22, 2024
7dc29b3
Cover JSON null and literal null string in unit test case
mattlord Dec 22, 2024
79d3f08
Update unit test for SQL conversion
mattlord Dec 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/cluster_endtoend_onlineddl_vrepl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard onlineddl_vrepl | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard onlineddl_vrepl_stress | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard onlineddl_vrepl_stress_suite | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard onlineddl_vrepl_suite | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/cluster_endtoend_schemadiff_vrepl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard schemadiff_vrepl | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_across_db_versions | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/cluster_endtoend_vreplication_basic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_basic | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_cellalias | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_copy_parallel | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_foreign_key_stress | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_mariadb_to_mysql | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/cluster_endtoend_vreplication_migrate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_migrate | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_multi_tenant | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_partial_movetables_and_materialize | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/cluster_endtoend_vreplication_v2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_v2 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ jobs:
binlog-transaction-compression=ON
EOF
cat <<-EOF>>./config/mycnf/mysql8026.cnf
binlog-row-value-options=PARTIAL_JSON
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard vreplication_vtctldclient_vdiff2_movetables_tz | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ jobs:
- name: Upload coverage reports to codecov.io
if: steps.changes.outputs.changed_files == 'true'
uses: codecov/codecov-action@v4
uses: codecov/codecov-action@015f24e6818733317a2da2edd6290ab26238649a # https://github.com/codecov/codecov-action/releases/tag/v5.0.7
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't help with the periodic failures, but good to upgrade anyway as the latest GA is 5.1.

with:
fail_ci_if_error: true
verbose: true
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_evalengine_mysql57.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="1"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql57"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_evalengine_mysql80.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="1"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql80"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_evalengine_mysql84.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="1"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql84"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_mysql57.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="0"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql57"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_mysql80.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="0"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql80"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/unit_test_mysql84.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,9 @@ jobs:
export NOVTADMINBUILD=1
export VTEVALENGINETEST="0"
# We sometimes need to alter the behavior based on the platform we're
# testing, e.g. MySQL 5.7 vs 8.0.
export CI_DB_PLATFORM="mysql84"
eatmydata -- make unit_test | tee -a output.txt | go-junit-report -set-exit-code > report.xml
Expand Down
94 changes: 92 additions & 2 deletions go/mysql/binlog/binlog_json.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ limitations under the License.
package binlog

import (
"bytes"
"encoding/binary"
"fmt"
"math"
Expand All @@ -25,9 +26,12 @@ import (
"vitess.io/vitess/go/hack"
"vitess.io/vitess/go/mysql/format"
"vitess.io/vitess/go/mysql/json"
"vitess.io/vitess/go/sqltypes"
"vitess.io/vitess/go/vt/sqlparser"
"vitess.io/vitess/go/vt/vterrors"

querypb "vitess.io/vitess/go/vt/proto/query"
vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
"vitess.io/vitess/go/vt/vterrors"
)

/*
Expand All @@ -44,6 +48,14 @@ https://github.com/shyiko/mysql-binlog-connector-java/pull/119/files
https://github.com/noplay/python-mysql-replication/blob/175df28cc8b536a68522ff9b09dc5440adad6094/pymysqlreplication/packet.py
*/

type jsonDiffOp uint8

const (
jsonDiffOpReplace = jsonDiffOp(0)
jsonDiffOpInsert = jsonDiffOp(1)
jsonDiffOpRemove = jsonDiffOp(2)
)

// ParseBinaryJSON provides the parsing function from the mysql binary json
// representation to a JSON value instance.
func ParseBinaryJSON(data []byte) (*json.Value, error) {
Expand All @@ -60,6 +72,84 @@ func ParseBinaryJSON(data []byte) (*json.Value, error) {
return node, nil
}

// ParseBinaryJSONDiff provides the parsing function from the binary MySQL
// JSON diff representation to an SQL expression.
func ParseBinaryJSONDiff(data []byte) (sqltypes.Value, error) {
diff := bytes.Buffer{}
// Reasonable estimate of the space we'll need to build the SQL
// expression in order to try and avoid reallocations w/o
// overallocating too much.
diff.Grow(int(float32(len(data)) * 1.25))
pos := 0
outer := false
innerStr := ""

for pos < len(data) {
opType := jsonDiffOp(data[pos])
pos++
if outer {
innerStr = diff.String()
diff.Reset()
}
switch opType {
case jsonDiffOpReplace:
diff.WriteString("JSON_REPLACE(")
case jsonDiffOpInsert:
diff.WriteString("JSON_INSERT(")
case jsonDiffOpRemove:
diff.WriteString("JSON_REMOVE(")
default:
// Can be a JSON null.
js, err := ParseBinaryJSON(data)
if err == nil && js.Type() == json.TypeNull {
return sqltypes.MakeTrusted(sqltypes.Expression, js.MarshalTo(nil)), nil
}
return sqltypes.Value{}, vterrors.Errorf(vtrpcpb.Code_INVALID_ARGUMENT,
"invalid JSON diff operation: %d", opType)
}
if outer {
diff.WriteString(innerStr)
diff.WriteString(", ")
} else { // Only the inner most function has the field name
diff.WriteString("%s, ") // This will later be replaced by the field name
}
outer = true

pathLen, readTo := readVariableLength(data, pos)
pos = readTo
path := data[pos : pos+pathLen]
pos += pathLen
// We have to specify the unicode character set for the strings we
// use in the expression as the connection can be using a different
// character set (e.g. vreplication always uses set names binary).
diff.WriteString(sqlparser.Utf8mb4Str)
diff.WriteByte('\'')
diff.Write(path)
diff.WriteByte('\'')
if opType == jsonDiffOpRemove { // No value for remove
diff.WriteByte(')')
continue
}

diff.WriteString(", ")
valueLen, readTo := readVariableLength(data, pos)
pos = readTo
value, err := ParseBinaryJSON(data[pos : pos+valueLen])
if err != nil {
return sqltypes.Value{}, vterrors.Errorf(vtrpcpb.Code_INVALID_ARGUMENT,
"cannot read JSON diff value for path %s: %v", path, err)
}
pos += valueLen
if value.Type() == json.TypeString {
diff.WriteString(sqlparser.Utf8mb4Str)
}
diff.Write(value.MarshalTo(nil))
diff.WriteByte(')')
}

return sqltypes.MakeTrusted(sqltypes.Expression, diff.Bytes()), nil
}

// jsonDataType has the values used in the mysql json binary representation to denote types.
// We have string, literal(true/false/null), number, object or array types.
// large object => doc size > 64K: you get pointers instead of inline values.
Expand Down Expand Up @@ -315,7 +405,7 @@ func binparserOpaque(_ jsonDataType, data []byte, pos int) (node *json.Value, er
precision := decimalData[0]
scale := decimalData[1]
metadata := (uint16(precision) << 8) + uint16(scale)
val, _, err := CellValue(decimalData, 2, TypeNewDecimal, metadata, &querypb.Field{Type: querypb.Type_DECIMAL})
val, _, err := CellValue(decimalData, 2, TypeNewDecimal, metadata, &querypb.Field{Type: querypb.Type_DECIMAL}, false)
if err != nil {
return nil, err
}
Expand Down
Loading
Loading