-
-
Notifications
You must be signed in to change notification settings - Fork 129
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
DistinctKeyVisitor and LogicalPlanDistinctKeys logical operators
- Loading branch information
1 parent
c071c93
commit 13c984b
Showing
5 changed files
with
107 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
# DistinctKeyVisitor | ||
|
||
`DistinctKeyVisitor` is a [LogicalPlanVisitor](cost-based-optimization/LogicalPlanVisitor.md) (of [Expression](expressions/Expression.md)s). | ||
|
||
`DistinctKeyVisitor` is used when a [logical operator](logical-operators/LogicalPlanDistinctKeys.md) is requested for the [distinct keys](#distinctKeys) with [spark.sql.optimizer.propagateDistinctKeys.enabled](configuration-properties.md#spark.sql.optimizer.propagateDistinctKeys.enabled) configuration property enabled. | ||
|
||
??? note "Singleton Object" | ||
`DistinctKeyVisitor` is a Scala **object** which is a class that has exactly one instance. It is created lazily when it is referenced, like a lazy val. | ||
|
||
Learn more in [Tour of Scala](https://docs.scala-lang.org/tour/singleton-objects.html). | ||
|
||
## visitAggregate { #visitAggregate } | ||
|
||
??? note "LogicalPlanVisitor" | ||
|
||
```scala | ||
visitAggregate( | ||
p: Aggregate): Set[ExpressionSet] | ||
``` | ||
|
||
`visitAggregate` is part of the [LogicalPlanVisitor](cost-based-optimization/LogicalPlanVisitor.md#visitAggregate) abstraction. | ||
|
||
`visitAggregate`...FIXME | ||
|
||
## visitJoin { #visitJoin } | ||
|
||
??? note "LogicalPlanVisitor" | ||
|
||
```scala | ||
visitJoin( | ||
p: Join): Set[ExpressionSet] | ||
``` | ||
|
||
`visitJoin` is part of the [LogicalPlanVisitor](cost-based-optimization/LogicalPlanVisitor.md#visitJoin) abstraction. | ||
|
||
`visitJoin`...FIXME | ||
|
||
## visitOffset { #visitOffset } | ||
|
||
??? note "LogicalPlanVisitor" | ||
|
||
```scala | ||
visitOffset( | ||
p: Offset): Set[ExpressionSet] | ||
``` | ||
|
||
`visitOffset` is part of the [LogicalPlanVisitor](cost-based-optimization/LogicalPlanVisitor.md#visitOffset) abstraction. | ||
|
||
`visitOffset`...FIXME |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
--- | ||
title: LogicalPlanDistinctKeys | ||
--- | ||
|
||
# LogicalPlanDistinctKeys Logical Operators | ||
|
||
`LogicalPlanDistinctKeys` is an extension of the [LogicalPlan](LogicalPlan.md) abstraction for logical operators that know their [distinct keys](#distinctKeys). | ||
|
||
All [logical operators](LogicalPlan.md) are `LogicalPlanDistinctKeys`. | ||
|
||
## Distinct Keys { #distinctKeys } | ||
|
||
```scala | ||
distinctKeys: Set[ExpressionSet] | ||
``` | ||
|
||
??? note "Lazy Value" | ||
`distinctKeys` is a Scala **lazy value** to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards. | ||
|
||
Learn more in the [Scala Language Specification]({{ scala.spec }}/05-classes-and-objects.html#lazy). | ||
|
||
`distinctKeys` uses [DistinctKeyVisitor](../DistinctKeyVisitor.md) to [determine the distinct keys](../cost-based-optimization/LogicalPlanVisitor.md#visit) of this [logical operator](LogicalPlan.md) when [spark.sql.optimizer.propagateDistinctKeys.enabled](../configuration-properties.md#spark.sql.optimizer.propagateDistinctKeys.enabled) configuration property is enabled. | ||
|
||
Otherwise, `distinctKeys` is always empty. | ||
|
||
--- | ||
|
||
`distinctKeys` is used when: | ||
|
||
* `EliminateOuterJoin` logical optimization is executed | ||
* `EliminateDistinct` logical optimization is executed | ||
* `RemoveRedundantAggregates` logical optimization is executed | ||
* `JoinEstimation` is requested to [estimateInnerOuterJoin](../cost-based-optimization/JoinEstimation.md#estimateInnerOuterJoin) | ||
* `SizeInBytesOnlyStatsPlanVisitor` is requested to [visitJoin](../cost-based-optimization/SizeInBytesOnlyStatsPlanVisitor.md#visitJoin) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters