From 9d1b1a298949b4e1da267504ec5feeb4a9d5a2d7 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 27 Jul 2023 09:52:50 -0600 Subject: [PATCH 1/3] Update documentation to add Databricks 12.2 as a supported platform --- docs/additional-functionality/delta-lake-support.md | 4 +++- docs/additional-functionality/rapids-shuffle.md | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/additional-functionality/delta-lake-support.md b/docs/additional-functionality/delta-lake-support.md index 4cda7aaf661..1509a853b37 100644 --- a/docs/additional-functionality/delta-lake-support.md +++ b/docs/additional-functionality/delta-lake-support.md @@ -38,6 +38,7 @@ Delta Lake writes: - Delta Lake version 2.4.0 on Apache Spark 3.4.x - Delta Lake on Databricks 10.4 LTS - Delta Lake on Databricks 11.3 LTS +- Delta Lake on Databricks 12.2 LTS Delta Lake writes will not be accelerated on Spark 3.1.x or earlier. @@ -49,7 +50,8 @@ operation which is typically triggered via the DataFrame `write` API, e.g.: `data.write.format("delta").save(...)`. Table creation from selection, table insertion from SQL, and table merges are not currently -GPU accelerated. These operations will fallback to the CPU. +GPU accelerated. These operations will fallback to the CPU. Writes against tables that have +deletion vectors enabled will also fallback to the CPU. #### Automatic Optimization of Writes diff --git a/docs/additional-functionality/rapids-shuffle.md b/docs/additional-functionality/rapids-shuffle.md index 08252fdbbd7..4f42189bdde 100644 --- a/docs/additional-functionality/rapids-shuffle.md +++ b/docs/additional-functionality/rapids-shuffle.md @@ -32,6 +32,7 @@ in our plugin: | 3.4.1 | com.nvidia.spark.rapids.spark341.RapidsShuffleManager | | Databricks 10.4 | com.nvidia.spark.rapids.spark321db.RapidsShuffleManager | | Databricks 11.3 | com.nvidia.spark.rapids.spark330db.RapidsShuffleManager | +| Databricks 12.2 | com.nvidia.spark.rapids.spark332db.RapidsShuffleManager | ## Multi-Threaded Mode From 4c0ee796a15b5dac24297fedce440d00911cc3bc Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Thu, 27 Jul 2023 09:53:56 -0600 Subject: [PATCH 2/3] signoff Signed-off-by: Andy Grove From 7b9f86f3944473dda40485ed7b8c67e5c9c67a5b Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 28 Jul 2023 08:50:40 -0600 Subject: [PATCH 3/3] Add note about reads against tables with deletion vectors falling back to CPU --- docs/additional-functionality/delta-lake-support.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/additional-functionality/delta-lake-support.md b/docs/additional-functionality/delta-lake-support.md index 1509a853b37..724b63337c7 100644 --- a/docs/additional-functionality/delta-lake-support.md +++ b/docs/additional-functionality/delta-lake-support.md @@ -17,13 +17,14 @@ This document details the Delta Lake features that are supported. Delta Lake scans of the underlying Parquet files are presented in the query as normal Parquet reads, so the Parquet reads will be accelerated in the same way raw Parquet file reads are -accelerated. +accelerated. Reads against tables that have deletion vectors enabled will fallback to the CPU. ### Metadata Queries Reads of Delta Lake metadata, i.e.: the Delta log detailing the history of snapshots, will not be GPU accelerated. The CPU will continue to process metadata queries on Delta Lake tables. + ## Writing Delta Lake Tables Delta Lake write acceleration is enabled by default. To disable acceleration of Delta Lake