Add support for common table expressions

[systay]

Description

ANSI SQL has had support for WITH and common table expressions since SQL-99, and MySQL added support in the 8.0 release, so it's about time Vitess supports it.

This PR adds tentative support for WITH and CTEs. The approach I have taken is to rewrite the CTE into derived tables before the actual planning starts.

In this PR, only SELECT and UNION support WITH.

Related Issue(s)

Part of solving #4099

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[frouioui]

Can't we simplify this:

Suggested change

	"Query": "select id, foo, weight_string(id), weight_string(foo) from (select id, foo from (select id, foo from `user`) as x union select id, foo from (select id, foo from `user`) as x) as dt",
	"Query": "select id, foo, weight_string(id), weight_string(foo) from (select id, foo from `user`) as x union select id, foo, weight_string(id), weight_string(foo) from (select id, foo from `user`) as x"

[frouioui]

Any plan on supporting this? Also, is the case where the table name is prefixed with the database identifier unsupported? (i.e.: with user as (select aa from user where user.id=1) select ref.col from ref join my_db.user)

[systay]

yeah, long term I think we want much better support for CTEs. instead of rewriting them to derived tables they should be first hand citizens, so that the parts of a query that we push down to MySQL uses CTE if the original query used a CTE, and even supporting recursive CTEs using a new engine primitive

[frouioui]

that makes sense!

[frouioui]

Left a few questions, nits and suggestions. Approving this for now as there is nothing major blocking this for me. Really excited about this, thank you 🚀

[frouioui]

Still looks good to me after the last few changes.

[harshit-gangal]

nit: can all the cases have comments about what is getting tested?

[GuptaManan100]

lgtm

[harshit-gangal]

the right of join route can be improvement as it is using sharding key in where condition.

{
          "QueryType": "SELECT",
          "Original": "with x as (select * from authoritative) select music.id, x.col1 from music join x on music.id = x.user_id",
          "Instructions": {
            "OperatorType": "Join",
            "Variant": "Join",
            "JoinColumnIndexes": "L:0,R:0",
            "JoinVars": {
              "music_id": 0
            },
            "TableName": "music_authoritative",
            "Inputs": [
              {
                "OperatorType": "Route",
                "Variant": "Scatter",
                "Keyspace": {
                  "Name": "user",
                  "Sharded": true
                },
                "FieldQuery": "select music.id from music where 1 != 1",
                "Query": "select music.id from music",
                "Table": "music"
              },
              {
                "OperatorType": "Route",
                "Variant": "Scatter",
                "Keyspace": {
                  "Name": "user",
                  "Sharded": true
                },
                "FieldQuery": "select x.col1 from (select user_id, col1, col2 from authoritative where 1 != 1) as x where 1 != 1",
                "Query": "select x.col1 from (select user_id, col1, col2 from authoritative where user_id = :music_id) as x",
                "Table": "authoritative"
              }
            ]
          },
          "TablesUsed": [
            "user.authoritative",
            "user.music"
          ]
        }