[BugFix] Fix ambiguous exception when type mismatch in fillnull command #960
Conversation
Signed-off-by: Heng Qian <[email protected]>
qianheng-aws
requested review from
dai-chen,
mengweieric,
vmmusings,
penghuo,
seankao-az,
anirudha,
kaituo,
YANG-DB,
noCharger,
LantaoJin and
ykmr1224
as code owners
| @@ -452,10 +453,30 @@ public LogicalPlan visitFillNull(FillNull fillNull, CatalystPlanContext context) | |||
| Seq<NamedExpression> projectExpressions = context.retainAllNamedParseExpressions(p -> (NamedExpression) p); | |||
| // build the plan with the projection step | |||
| context.apply(p -> new org.apache.spark.sql.catalyst.plans.logical.Project(projectExpressions, p)); | |||
| LogicalPlan resultWithoutDuplicatedColumns = context.apply(logicalPlan -> DataFrameDropColumns$.MODULE$.apply(seq(toDrop), logicalPlan)); | |||
| LogicalPlan resultWithoutDuplicatedColumns = context.apply(dropOriginalColumns(p -> p.children().head(), toDrop)); | |||
There was a problem hiding this comment.
Could you test this patch with the latest Spark version locally to double confirm as a long term fixing?
There was a problem hiding this comment.
Verified with Spark branch-3.5, which includes this fix: apache/spark#48240.
As expected, it will throw ambiguous exception for all cases and could be addressed with this PR.
|
@qianheng-aws @LantaoJin What is the expected GA spark version we can assume this to be released with (~3.5 ? ) ? |
This patch should be released in Spark v3.5.4 and v4.0.0. I have no idea about the date of release. I think this PR resolves the ambiguous issue so we don't need a notice in release any more. |
…nd (opensearch-project#960) Signed-off-by: Heng Qian <[email protected]>
Description
Fix ambiguous exception when type mismatch in fillnull command
Related Issues
Resolves #959
Check List
--signoffBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.