Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support recursive CTEs #16427

Merged
merged 29 commits into from
Aug 19, 2024
Merged

Support recursive CTEs #16427

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c26bd04
Semantic analysis for recursive CTEs
systay Jul 18, 2024
08b10c4
feat: add recurse operator and start of CTE planning
systay Jul 19, 2024
c32fb87
first steps in adding Recurse primitive
systay Jul 22, 2024
55dfef9
remove subquery from cte
systay Jul 25, 2024
09aa5d8
check for invalid CTEs
systay Jul 29, 2024
9e8f5ba
feat: use the new CTE field on RealTable
systay Jul 30, 2024
c38ae57
feat: handle star expansion on CTEs
systay Jul 30, 2024
705c079
first plan with recursion at the vtgate level passes
systay Jul 31, 2024
adedb5b
change terminology for recursice ctes to seed/term
systay Jul 31, 2024
6c9ca09
codegen
systay Jul 31, 2024
c6277bb
panic on the error, no the string
systay Jul 31, 2024
426aa01
mark query as single unsharded keyspace when possible
systay Aug 1, 2024
94d8fd5
move test
systay Aug 1, 2024
8fd269a
added end-to-end tests for recursive queries
systay Aug 5, 2024
5746dba
also handle CTE on the LHS of joins
systay Aug 5, 2024
def7c28
make sure to mark CTE as recursive
systay Aug 5, 2024
c7470c5
rewrite expressions passing into ctes
systay Aug 5, 2024
b6e04a5
handle projections on the term using expressions from the seed
systay Aug 7, 2024
730c3b5
handle projections on merged CTEs
systay Aug 7, 2024
a2ef673
add more planner tests
systay Aug 7, 2024
2b4088e
test: add cte end to end tests
systay Aug 8, 2024
c207131
handle recursive CTE table IDs correctly
systay Aug 8, 2024
8566ebe
preserve CTE table alias
systay Aug 8, 2024
5f2aa27
changelog
systay Aug 14, 2024
efda1df
handle ws offsets on recursive ctes
systay Aug 14, 2024
f8501c0
remove unused code
systay Aug 14, 2024
3aaad01
make sure to respect the transaction needs
systay Aug 15, 2024
36a458d
address review comments
systay Aug 19, 2024
b67eb1a
Merge remote-tracking branch 'upstream/main' into recursive-cte
systay Aug 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,4 @@ report

# mise files
.mise.toml
/errors/
4 changes: 4 additions & 0 deletions changelog/21.0/21.0.0/summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- **[New VTGate Shutdown Behavior](#new-vtgate-shutdown-behavior)**
- **[Tablet Throttler: Multi-Metric support](#tablet-throttler)**
- **[Allow Cross Cell Promotion in PRS](#allow-cross-cell)**
- **[Support for recursive CTEs](#recursive-cte)**

## <a id="major-changes"/>Major Changes

Expand Down Expand Up @@ -102,3 +103,6 @@ Metrics are assigned a default _scope_, which could be `self` (isolated to the t
Up until now if the users wanted to promote a replica in a different cell than the current primary using `PlannedReparentShard`, they had to specify the new primary with the `--new-primary` flag.

We have now added a new flag `--allow-cross-cell-promotion` that lets `PlannedReparentShard` choose a primary in a different cell even if no new primary is provided explicitly.

### <a id="recursive-cte"/>Experimental support for recursive CTEs
We have added experimental support for recursive CTEs in Vitess. We are marking it as experimental because it is not yet fully tested and may have some limitations. We are looking for feedback from the community to improve this feature.
47 changes: 26 additions & 21 deletions go/mysql/sqlerror/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,27 +255,32 @@ const (
ERJSONValueTooBig = ErrorCode(3150)
ERJSONDocumentTooDeep = ErrorCode(3157)

ERLockNowait = ErrorCode(3572)
ERRegexpStringNotTerminated = ErrorCode(3684)
ERRegexpBufferOverflow = ErrorCode(3684)
ERRegexpIllegalArgument = ErrorCode(3685)
ERRegexpIndexOutOfBounds = ErrorCode(3686)
ERRegexpInternal = ErrorCode(3687)
ERRegexpRuleSyntax = ErrorCode(3688)
ERRegexpBadEscapeSequence = ErrorCode(3689)
ERRegexpUnimplemented = ErrorCode(3690)
ERRegexpMismatchParen = ErrorCode(3691)
ERRegexpBadInterval = ErrorCode(3692)
ERRRegexpMaxLtMin = ErrorCode(3693)
ERRegexpInvalidBackRef = ErrorCode(3694)
ERRegexpLookBehindLimit = ErrorCode(3695)
ERRegexpMissingCloseBracket = ErrorCode(3696)
ERRegexpInvalidRange = ErrorCode(3697)
ERRegexpStackOverflow = ErrorCode(3698)
ERRegexpTimeOut = ErrorCode(3699)
ERRegexpPatternTooBig = ErrorCode(3700)
ERRegexpInvalidCaptureGroup = ErrorCode(3887)
ERRegexpInvalidFlag = ErrorCode(3900)
ERLockNowait = ErrorCode(3572)
ERCTERecursiveRequiresUnion = ErrorCode(3573)
ERCTERecursiveForbidsAggregation = ErrorCode(3575)
ERCTERecursiveForbiddenJoinOrder = ErrorCode(3576)
ERCTERecursiveRequiresSingleReference = ErrorCode(3577)
ERCTEMaxRecursionDepth = ErrorCode(3636)
ERRegexpStringNotTerminated = ErrorCode(3684)
ERRegexpBufferOverflow = ErrorCode(3684)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error code 3684 is duplicated. Is that an error?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the same on the MySQL manual. I think the duplication in Vitess it's intentional 🤷

Also, the whole page is about errors, so clearly it's an error. 😜

ERRegexpIllegalArgument = ErrorCode(3685)
ERRegexpIndexOutOfBounds = ErrorCode(3686)
ERRegexpInternal = ErrorCode(3687)
ERRegexpRuleSyntax = ErrorCode(3688)
ERRegexpBadEscapeSequence = ErrorCode(3689)
ERRegexpUnimplemented = ErrorCode(3690)
ERRegexpMismatchParen = ErrorCode(3691)
ERRegexpBadInterval = ErrorCode(3692)
ERRRegexpMaxLtMin = ErrorCode(3693)
ERRegexpInvalidBackRef = ErrorCode(3694)
ERRegexpLookBehindLimit = ErrorCode(3695)
ERRegexpMissingCloseBracket = ErrorCode(3696)
ERRegexpInvalidRange = ErrorCode(3697)
ERRegexpStackOverflow = ErrorCode(3698)
ERRegexpTimeOut = ErrorCode(3699)
ERRegexpPatternTooBig = ErrorCode(3700)
ERRegexpInvalidCaptureGroup = ErrorCode(3887)
ERRegexpInvalidFlag = ErrorCode(3900)

ERCharacterSetMismatch = ErrorCode(3995)

Expand Down
153 changes: 79 additions & 74 deletions go/mysql/sqlerror/sql_error.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,80 +172,85 @@ type mysqlCode struct {
}

var stateToMysqlCode = map[vterrors.State]mysqlCode{
vterrors.Undefined: {num: ERUnknownError, state: SSUnknownSQLState},
vterrors.AccessDeniedError: {num: ERAccessDeniedError, state: SSAccessDeniedError},
vterrors.BadDb: {num: ERBadDb, state: SSClientError},
vterrors.BadFieldError: {num: ERBadFieldError, state: SSBadFieldError},
vterrors.BadTableError: {num: ERBadTable, state: SSUnknownTable},
vterrors.CantUseOptionHere: {num: ERCantUseOptionHere, state: SSClientError},
vterrors.DataOutOfRange: {num: ERDataOutOfRange, state: SSDataOutOfRange},
vterrors.DbCreateExists: {num: ERDbCreateExists, state: SSUnknownSQLState},
vterrors.DbDropExists: {num: ERDbDropExists, state: SSUnknownSQLState},
vterrors.DupFieldName: {num: ERDupFieldName, state: SSDupFieldName},
vterrors.EmptyQuery: {num: EREmptyQuery, state: SSClientError},
vterrors.IncorrectGlobalLocalVar: {num: ERIncorrectGlobalLocalVar, state: SSUnknownSQLState},
vterrors.InnodbReadOnly: {num: ERInnodbReadOnly, state: SSUnknownSQLState},
vterrors.LockOrActiveTransaction: {num: ERLockOrActiveTransaction, state: SSUnknownSQLState},
vterrors.NoDB: {num: ERNoDb, state: SSNoDB},
vterrors.NoSuchTable: {num: ERNoSuchTable, state: SSUnknownTable},
vterrors.NotSupportedYet: {num: ERNotSupportedYet, state: SSClientError},
vterrors.ForbidSchemaChange: {num: ERForbidSchemaChange, state: SSUnknownSQLState},
vterrors.MixOfGroupFuncAndFields: {num: ERMixOfGroupFuncAndFields, state: SSClientError},
vterrors.NetPacketTooLarge: {num: ERNetPacketTooLarge, state: SSNetError},
vterrors.NonUniqError: {num: ERNonUniq, state: SSConstraintViolation},
vterrors.NonUniqTable: {num: ERNonUniqTable, state: SSClientError},
vterrors.NonUpdateableTable: {num: ERNonUpdateableTable, state: SSUnknownSQLState},
vterrors.QueryInterrupted: {num: ERQueryInterrupted, state: SSQueryInterrupted},
vterrors.SPDoesNotExist: {num: ERSPDoesNotExist, state: SSClientError},
vterrors.SyntaxError: {num: ERSyntaxError, state: SSClientError},
vterrors.UnsupportedPS: {num: ERUnsupportedPS, state: SSUnknownSQLState},
vterrors.UnknownSystemVariable: {num: ERUnknownSystemVariable, state: SSUnknownSQLState},
vterrors.UnknownTable: {num: ERUnknownTable, state: SSUnknownTable},
vterrors.WrongGroupField: {num: ERWrongGroupField, state: SSClientError},
vterrors.WrongNumberOfColumnsInSelect: {num: ERWrongNumberOfColumnsInSelect, state: SSWrongNumberOfColumns},
vterrors.WrongTypeForVar: {num: ERWrongTypeForVar, state: SSClientError},
vterrors.WrongValueForVar: {num: ERWrongValueForVar, state: SSClientError},
vterrors.WrongValue: {num: ERWrongValue, state: SSUnknownSQLState},
vterrors.WrongFieldWithGroup: {num: ERWrongFieldWithGroup, state: SSClientError},
vterrors.ServerNotAvailable: {num: ERServerIsntAvailable, state: SSNetError},
vterrors.CantDoThisInTransaction: {num: ERCantDoThisDuringAnTransaction, state: SSCantDoThisDuringAnTransaction},
vterrors.RequiresPrimaryKey: {num: ERRequiresPrimaryKey, state: SSClientError},
vterrors.RowIsReferenced2: {num: ERRowIsReferenced2, state: SSConstraintViolation},
vterrors.NoReferencedRow2: {num: ErNoReferencedRow2, state: SSConstraintViolation},
vterrors.NoSuchSession: {num: ERUnknownComError, state: SSNetError},
vterrors.OperandColumns: {num: EROperandColumns, state: SSWrongNumberOfColumns},
vterrors.WrongValueCountOnRow: {num: ERWrongValueCountOnRow, state: SSWrongValueCountOnRow},
vterrors.WrongArguments: {num: ERWrongArguments, state: SSUnknownSQLState},
vterrors.ViewWrongList: {num: ERViewWrongList, state: SSUnknownSQLState},
vterrors.UnknownStmtHandler: {num: ERUnknownStmtHandler, state: SSUnknownSQLState},
vterrors.KeyDoesNotExist: {num: ERKeyDoesNotExist, state: SSClientError},
vterrors.UnknownTimeZone: {num: ERUnknownTimeZone, state: SSUnknownSQLState},
vterrors.RegexpStringNotTerminated: {num: ERRegexpStringNotTerminated, state: SSUnknownSQLState},
vterrors.RegexpBufferOverflow: {num: ERRegexpBufferOverflow, state: SSUnknownSQLState},
vterrors.RegexpIllegalArgument: {num: ERRegexpIllegalArgument, state: SSUnknownSQLState},
vterrors.RegexpIndexOutOfBounds: {num: ERRegexpIndexOutOfBounds, state: SSUnknownSQLState},
vterrors.RegexpInternal: {num: ERRegexpInternal, state: SSUnknownSQLState},
vterrors.RegexpRuleSyntax: {num: ERRegexpRuleSyntax, state: SSUnknownSQLState},
vterrors.RegexpBadEscapeSequence: {num: ERRegexpBadEscapeSequence, state: SSUnknownSQLState},
vterrors.RegexpUnimplemented: {num: ERRegexpUnimplemented, state: SSUnknownSQLState},
vterrors.RegexpMismatchParen: {num: ERRegexpMismatchParen, state: SSUnknownSQLState},
vterrors.RegexpBadInterval: {num: ERRegexpBadInterval, state: SSUnknownSQLState},
vterrors.RegexpMaxLtMin: {num: ERRRegexpMaxLtMin, state: SSUnknownSQLState},
vterrors.RegexpInvalidBackRef: {num: ERRegexpInvalidBackRef, state: SSUnknownSQLState},
vterrors.RegexpLookBehindLimit: {num: ERRegexpLookBehindLimit, state: SSUnknownSQLState},
vterrors.RegexpMissingCloseBracket: {num: ERRegexpMissingCloseBracket, state: SSUnknownSQLState},
vterrors.RegexpInvalidRange: {num: ERRegexpInvalidRange, state: SSUnknownSQLState},
vterrors.RegexpStackOverflow: {num: ERRegexpStackOverflow, state: SSUnknownSQLState},
vterrors.RegexpTimeOut: {num: ERRegexpTimeOut, state: SSUnknownSQLState},
vterrors.RegexpPatternTooBig: {num: ERRegexpPatternTooBig, state: SSUnknownSQLState},
vterrors.RegexpInvalidFlag: {num: ERRegexpInvalidFlag, state: SSUnknownSQLState},
vterrors.RegexpInvalidCaptureGroup: {num: ERRegexpInvalidCaptureGroup, state: SSUnknownSQLState},
vterrors.CharacterSetMismatch: {num: ERCharacterSetMismatch, state: SSUnknownSQLState},
vterrors.WrongParametersToNativeFct: {num: ERWrongParametersToNativeFct, state: SSUnknownSQLState},
vterrors.KillDeniedError: {num: ERKillDenied, state: SSUnknownSQLState},
vterrors.BadNullError: {num: ERBadNullError, state: SSConstraintViolation},
vterrors.InvalidGroupFuncUse: {num: ERInvalidGroupFuncUse, state: SSUnknownSQLState},
vterrors.VectorConversion: {num: ERVectorConversion, state: SSUnknownSQLState},
vterrors.Undefined: {num: ERUnknownError, state: SSUnknownSQLState},
vterrors.AccessDeniedError: {num: ERAccessDeniedError, state: SSAccessDeniedError},
vterrors.BadDb: {num: ERBadDb, state: SSClientError},
vterrors.BadFieldError: {num: ERBadFieldError, state: SSBadFieldError},
vterrors.BadTableError: {num: ERBadTable, state: SSUnknownTable},
vterrors.CantUseOptionHere: {num: ERCantUseOptionHere, state: SSClientError},
vterrors.DataOutOfRange: {num: ERDataOutOfRange, state: SSDataOutOfRange},
vterrors.DbCreateExists: {num: ERDbCreateExists, state: SSUnknownSQLState},
vterrors.DbDropExists: {num: ERDbDropExists, state: SSUnknownSQLState},
vterrors.DupFieldName: {num: ERDupFieldName, state: SSDupFieldName},
vterrors.EmptyQuery: {num: EREmptyQuery, state: SSClientError},
vterrors.IncorrectGlobalLocalVar: {num: ERIncorrectGlobalLocalVar, state: SSUnknownSQLState},
vterrors.InnodbReadOnly: {num: ERInnodbReadOnly, state: SSUnknownSQLState},
vterrors.LockOrActiveTransaction: {num: ERLockOrActiveTransaction, state: SSUnknownSQLState},
vterrors.NoDB: {num: ERNoDb, state: SSNoDB},
vterrors.NoSuchTable: {num: ERNoSuchTable, state: SSUnknownTable},
vterrors.NotSupportedYet: {num: ERNotSupportedYet, state: SSClientError},
vterrors.ForbidSchemaChange: {num: ERForbidSchemaChange, state: SSUnknownSQLState},
vterrors.MixOfGroupFuncAndFields: {num: ERMixOfGroupFuncAndFields, state: SSClientError},
vterrors.NetPacketTooLarge: {num: ERNetPacketTooLarge, state: SSNetError},
vterrors.NonUniqError: {num: ERNonUniq, state: SSConstraintViolation},
vterrors.NonUniqTable: {num: ERNonUniqTable, state: SSClientError},
vterrors.NonUpdateableTable: {num: ERNonUpdateableTable, state: SSUnknownSQLState},
vterrors.QueryInterrupted: {num: ERQueryInterrupted, state: SSQueryInterrupted},
vterrors.SPDoesNotExist: {num: ERSPDoesNotExist, state: SSClientError},
vterrors.SyntaxError: {num: ERSyntaxError, state: SSClientError},
vterrors.UnsupportedPS: {num: ERUnsupportedPS, state: SSUnknownSQLState},
vterrors.UnknownSystemVariable: {num: ERUnknownSystemVariable, state: SSUnknownSQLState},
vterrors.UnknownTable: {num: ERUnknownTable, state: SSUnknownTable},
vterrors.WrongGroupField: {num: ERWrongGroupField, state: SSClientError},
vterrors.WrongNumberOfColumnsInSelect: {num: ERWrongNumberOfColumnsInSelect, state: SSWrongNumberOfColumns},
vterrors.WrongTypeForVar: {num: ERWrongTypeForVar, state: SSClientError},
vterrors.WrongValueForVar: {num: ERWrongValueForVar, state: SSClientError},
vterrors.WrongValue: {num: ERWrongValue, state: SSUnknownSQLState},
vterrors.WrongFieldWithGroup: {num: ERWrongFieldWithGroup, state: SSClientError},
vterrors.ServerNotAvailable: {num: ERServerIsntAvailable, state: SSNetError},
vterrors.CantDoThisInTransaction: {num: ERCantDoThisDuringAnTransaction, state: SSCantDoThisDuringAnTransaction},
vterrors.RequiresPrimaryKey: {num: ERRequiresPrimaryKey, state: SSClientError},
vterrors.RowIsReferenced2: {num: ERRowIsReferenced2, state: SSConstraintViolation},
vterrors.NoReferencedRow2: {num: ErNoReferencedRow2, state: SSConstraintViolation},
vterrors.NoSuchSession: {num: ERUnknownComError, state: SSNetError},
vterrors.OperandColumns: {num: EROperandColumns, state: SSWrongNumberOfColumns},
vterrors.WrongValueCountOnRow: {num: ERWrongValueCountOnRow, state: SSWrongValueCountOnRow},
vterrors.WrongArguments: {num: ERWrongArguments, state: SSUnknownSQLState},
vterrors.ViewWrongList: {num: ERViewWrongList, state: SSUnknownSQLState},
vterrors.UnknownStmtHandler: {num: ERUnknownStmtHandler, state: SSUnknownSQLState},
vterrors.KeyDoesNotExist: {num: ERKeyDoesNotExist, state: SSClientError},
vterrors.UnknownTimeZone: {num: ERUnknownTimeZone, state: SSUnknownSQLState},
vterrors.RegexpStringNotTerminated: {num: ERRegexpStringNotTerminated, state: SSUnknownSQLState},
vterrors.RegexpBufferOverflow: {num: ERRegexpBufferOverflow, state: SSUnknownSQLState},
vterrors.RegexpIllegalArgument: {num: ERRegexpIllegalArgument, state: SSUnknownSQLState},
vterrors.RegexpIndexOutOfBounds: {num: ERRegexpIndexOutOfBounds, state: SSUnknownSQLState},
vterrors.RegexpInternal: {num: ERRegexpInternal, state: SSUnknownSQLState},
vterrors.RegexpRuleSyntax: {num: ERRegexpRuleSyntax, state: SSUnknownSQLState},
vterrors.RegexpBadEscapeSequence: {num: ERRegexpBadEscapeSequence, state: SSUnknownSQLState},
vterrors.RegexpUnimplemented: {num: ERRegexpUnimplemented, state: SSUnknownSQLState},
vterrors.RegexpMismatchParen: {num: ERRegexpMismatchParen, state: SSUnknownSQLState},
vterrors.RegexpBadInterval: {num: ERRegexpBadInterval, state: SSUnknownSQLState},
vterrors.RegexpMaxLtMin: {num: ERRRegexpMaxLtMin, state: SSUnknownSQLState},
vterrors.RegexpInvalidBackRef: {num: ERRegexpInvalidBackRef, state: SSUnknownSQLState},
vterrors.RegexpLookBehindLimit: {num: ERRegexpLookBehindLimit, state: SSUnknownSQLState},
vterrors.RegexpMissingCloseBracket: {num: ERRegexpMissingCloseBracket, state: SSUnknownSQLState},
vterrors.RegexpInvalidRange: {num: ERRegexpInvalidRange, state: SSUnknownSQLState},
vterrors.RegexpStackOverflow: {num: ERRegexpStackOverflow, state: SSUnknownSQLState},
vterrors.RegexpTimeOut: {num: ERRegexpTimeOut, state: SSUnknownSQLState},
vterrors.RegexpPatternTooBig: {num: ERRegexpPatternTooBig, state: SSUnknownSQLState},
vterrors.RegexpInvalidFlag: {num: ERRegexpInvalidFlag, state: SSUnknownSQLState},
vterrors.RegexpInvalidCaptureGroup: {num: ERRegexpInvalidCaptureGroup, state: SSUnknownSQLState},
vterrors.CharacterSetMismatch: {num: ERCharacterSetMismatch, state: SSUnknownSQLState},
vterrors.WrongParametersToNativeFct: {num: ERWrongParametersToNativeFct, state: SSUnknownSQLState},
vterrors.KillDeniedError: {num: ERKillDenied, state: SSUnknownSQLState},
vterrors.BadNullError: {num: ERBadNullError, state: SSConstraintViolation},
vterrors.InvalidGroupFuncUse: {num: ERInvalidGroupFuncUse, state: SSUnknownSQLState},
vterrors.VectorConversion: {num: ERVectorConversion, state: SSUnknownSQLState},
vterrors.CTERecursiveRequiresSingleReference: {num: ERCTERecursiveRequiresSingleReference, state: SSUnknownSQLState},
vterrors.CTERecursiveRequiresUnion: {num: ERCTERecursiveRequiresUnion, state: SSUnknownSQLState},
vterrors.CTERecursiveForbidsAggregation: {num: ERCTERecursiveForbidsAggregation, state: SSUnknownSQLState},
vterrors.CTERecursiveForbiddenJoinOrder: {num: ERCTERecursiveForbiddenJoinOrder, state: SSUnknownSQLState},
vterrors.CTEMaxRecursionDepth: {num: ERCTEMaxRecursionDepth, state: SSUnknownSQLState},
}

func getStateToMySQLState(state vterrors.State) mysqlCode {
Expand Down
Loading
Loading