Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise hash joins #14644

Merged
merged 18 commits into from
Dec 12, 2023
Merged

Optimise hash joins #14644

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
be6480f
planner: skip projections when not needed
systay Nov 30, 2023
441c783
planner: split aggregation pushing into two files
systay Dec 4, 2023
7a3f53b
planbuilder error handling: let runtime.Error panics stay as panics
systay Dec 4, 2023
2dfe734
planbuilder refactor: prepare to have different hash joins pushed thr…
systay Dec 4, 2023
c292fe5
refactor: refactor to allow hash joins in aggregationg pushing
GuptaManan100 Dec 6, 2023
d1163ec
refactor: use hashColumns instead of the bare slice
systay Dec 7, 2023
9d37cc8
planner feat: push down aggregations through hash joins
systay Dec 7, 2023
0eba286
test: update tests
systay Dec 8, 2023
168a78d
refactor: use the joinColumns interface
systay Dec 8, 2023
818e7f2
feat: push grouping columns correctly through hash joins
systay Dec 8, 2023
7812f9a
refactor: clean up code
systay Dec 8, 2023
a41459a
test: add end to end tests
systay Dec 8, 2023
ac88590
refactor: rename method and clean up comment
systay Dec 8, 2023
f0f8ca0
Merge remote-tracking branch 'upstream/main' into hash-join-pushing
systay Dec 8, 2023
ca8dc6e
test: fix test expectation that got copied wrongly
systay Dec 8, 2023
35d84de
cleanup: delete unused code
systay Dec 9, 2023
8ae5b03
feat: push down distinct through hash joins
systay Dec 9, 2023
ad9d4e2
refactor: make sure we don't miss any cloning
systay Dec 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions go/test/endtoend/vtgate/queries/aggregation/aggregation_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,32 @@ func TestAggregateLeftJoin(t *testing.T) {
mcmp.AssertMatches("SELECT avg(t1.shardkey) FROM t1 LEFT JOIN t2 ON t1.t1_id = t2.id", `[[DECIMAL(0.5000)]]`)
mcmp.AssertMatches("SELECT avg(t2.shardkey) FROM t1 LEFT JOIN t2 ON t1.t1_id = t2.id", `[[DECIMAL(1.0000)]]`)
mcmp.AssertMatches("SELECT count(*) FROM t2 LEFT JOIN t1 ON t1.t1_id = t2.id WHERE IFNULL(t1.name, 'NOTSET') = 'r'", `[[INT64(1)]]`)

aggregations := []string{
"count(t1.shardkey)",
"count(t2.shardkey)",
"sum(t1.shardkey)",
"sum(t2.shardkey)",
"avg(t1.shardkey)",
"avg(t2.shardkey)",
"count(*)",
}

grouping := []string{
"t1.t1_id",
"t1.shardKey",
"t1.value",
"t2.id",
"t2.shardKey",
}

// quickly construct a big number of left join aggregation queries that have to be executed using the hash join
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

for _, agg := range aggregations {
for _, gb := range grouping {
query := fmt.Sprintf("SELECT %s FROM t1 LEFT JOIN (select id, shardkey from t2 limit 100) as t2 ON t1.t1_id = t2.id group by %s", agg, gb)
mcmp.Exec(query)
}
}
}

// TestScalarAggregate tests validates that only count is returned and no additional field is returned.gst
Expand Down
Loading
Loading