Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-50364][SQL] Implement serialization for LocalDateTime type in …
…Row.jsonValue ### What changes were proposed in this pull request? In this Pull request, I propose to add a LocalDateTime serializer to the Row.jsonValue method ```scala case (d: LocalDateTime, _) => JString(timestampFormatter.format(d)) ``` In order to enable JSON serialization of _TimestampNTZType_ columns ### Why are the changes needed? Currently trying to serialize a Row containing a _TimestampNTZType_ column results in an error: ```java [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G org.apache.spark.SparkIllegalArgumentException: [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G at org.apache.spark.sql.Row.toJson$1(Row.scala:663) at org.apache.spark.sql.Row.toJson$1(Row.scala:651) at org.apache.spark.sql.Row.jsonValue(Row.scala:665) at org.apache.spark.sql.Row.jsonValue$(Row.scala:598) at org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:28) at org.apache.spark.sql.RowJsonSuite.$anonfun$testJson$1(RowJsonSuite.scala:41) ``` How to reproduce the issue: ```scala import org.apache.spark.sql.Row import java.time.LocalDateTime val r = Row.fromSeq(LocalDateTime.of(2024,8,10,12,33) :: Nil) r.json r.prettyJson ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tests were added to the already existing RowJsonSuite.scala class ### Was this patch authored or co-authored using generative AI tooling? No Closes #48905 from karim-ramadan/bugfix/LocalDateTime-support-in-Row-json. Authored-by: Karim Ramadan <[email protected]> Signed-off-by: Max Gekk <[email protected]>
- Loading branch information