Skip to content

Commit

Permalink
Updating docs (GoogleCloudDataproc#1089)
Browse files Browse the repository at this point in the history
  • Loading branch information
@davidrabinowitz
davidrabinowitz authored Oct 12, 2023
1 parent d04e74b commit d04164a
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 24 deletions.
13 changes: 7 additions & 6 deletions CHANGES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,17 @@

## Next

* Issue #453: fix comment handling in query
* PR #1061: Handle NPE case when reading BQ table with NUMERIC fields. Thanks @hayssams !
* Issue #144: allow writing Spark String to BQ TIME type
* Issue #867: Support writing with RangePartitioning
* Issue #1062: Adding dataproc job ID and UUID labels to BigQuery jobs
* Issue #1046: Add a way to disable map type support
* Added new connector, `spark-3.4-bigquery` aimed to be used in Spark 3.4 and above. This connector implements new APIs and capabilities provided by the Spark Data Source V2 API.
* PR #1008: Adding support to expose BigQuery metrics using Spark custom metrics API.
* PR #1038: Logical plan now shows the BigQuery table of DirectBigQueryRelation. Thanks @idc101 !
* PR #1058: View names will appear in query plan instead of the materialized table
* PR #1061: Handle NPE case when reading BQ table with NUMERIC fields. Thanks @hayssams !
* PR #1069: Support TimestampNTZ datatype in spark 3.4
* Issue #453: fix comment handling in query
* Issue #144: allow writing Spark String to BQ TIME type
* Issue #867: Support writing with RangePartitioning
* Issue #1046: Add a way to disable map type support
* Issue #1062: Adding dataproc job ID and UUID labels to BigQuery jobs

## 0.32.2 - 2023-08-07

Expand Down
40 changes: 22 additions & 18 deletions README-template.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ The latest version of the connector is publicly available in the following links

| version | Link |
|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 3.4 | `gs://spark-lib/bigquery/spark-3.4-bigquery-${next-release-tag}.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.4-bigquery-${next-release-tag}.jar)) |
| Spark 3.3 | `gs://spark-lib/bigquery/spark-3.3-bigquery-${next-release-tag}.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.3-bigquery-${next-release-tag}.jar)) |
| Spark 3.2 | `gs://spark-lib/bigquery/spark-3.2-bigquery-${next-release-tag}.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.2-bigquery-${next-release-tag}.jar)) |
| Spark 3.1 | `gs://spark-lib/bigquery/spark-3.1-bigquery-${next-release-tag}.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.1-bigquery-${next-release-tag}.jar)) |
Expand All @@ -78,26 +79,28 @@ The final two connectors are Scala based connectors, please use the jar relevant
below.

### Connector to Spark Compatibility Matrix
| Connector \ Spark | 2.3 | 2.4<br>(Scala 2.11) | 2.4<br>(Scala 2.12) | 3.0 | 3.1 | 3.2 | 3.3 |
|---------------------------------------|---------|---------------------|---------------------|---------|---------|---------|---------|
| spark-3.3-bigquery | | | | | | | &check; |
| spark-3.2-bigquery | | | | | | &check; | &check; |
| spark-3.1-bigquery | | | | | &check; | &check; | &check; |
| spark-2.4-bigquery | | &check; | &check; | | | | |
| spark-bigquery-with-dependencies_2.13 | | | | | | &check; | &check; |
| spark-bigquery-with-dependencies_2.12 | | | &check; | &check; | &check; | &check; | &check; |
| spark-bigquery-with-dependencies_2.11 | &check; | &check; | | | | | |
| Connector \ Spark | 2.3 | 2.4 | 3.0 | 3.1 | 3.2 | 3.3 |3.4 |
|---------------------------------------|---------|---------|---------|---------|---------|---------|---------|
| spark-3.4-bigquery | | | | | | | &check; |
| spark-3.3-bigquery | | | | | | &check; | &check; |
| spark-3.2-bigquery | | | | | &check; | &check; | &check; |
| spark-3.1-bigquery | | | | &check; | &check; | &check; | &check; |
| spark-2.4-bigquery | | &check; | | | | | |
| spark-bigquery-with-dependencies_2.13 | | | | | &check; | &check; | &check; |
| spark-bigquery-with-dependencies_2.12 | | &check; | &check; | &check; | &check; | &check; | &check; |
| spark-bigquery-with-dependencies_2.11 | &check; | &check; | | | | | |

### Connector to Dataproc Image Compatibility Matrix
| Connector \ Dataproc Image | 1.3 | 1.4 | 1.5 | 2.0 | 2.1 | Serverless<br>Image 1.0 | Serverless<br>Image 2.0 |
|---------------------------------------|---------|---------|---------|---------|---------|-------------------------|-------------------------|
| spark-3.3-bigquery | | | | | &check; | &check; | &check; |
| spark-3.2-bigquery | | | | | &check; | &check; | &check; |
| spark-3.1-bigquery | | | | &check; | &check; | &check; | &check; |
| spark-2.4-bigquery | | &check; | &check; | | | | |
| spark-bigquery-with-dependencies_2.13 | | | | | | | &check; |
| spark-bigquery-with-dependencies_2.12 | | | &check; | &check; | &check; | &check; | |
| spark-bigquery-with-dependencies_2.11 | &check; | &check; | | | | | |
| Connector \ Dataproc Image | 1.3 | 1.4 | 1.5 | 2.0 | 2.1 | Serverless<br>Image 1.0 | Serverless<br>Image 2.0 | Serverless<br>Image 2.1 |
|---------------------------------------|---------|---------|---------|---------|---------|-------------------------|-------------------------|-------------------------|
| spark-3.4-bigquery | | | | | | | | &check; |
| spark-3.3-bigquery | | | | | &check; | &check; | &check; | &check; |
| spark-3.2-bigquery | | | | | &check; | &check; | &check; | &check; |
| spark-3.1-bigquery | | | | &check; | &check; | &check; | &check; | &check; |
| spark-2.4-bigquery | | &check; | &check; | | | | | |
| spark-bigquery-with-dependencies_2.13 | | | | | | | &check; | &check; |
| spark-bigquery-with-dependencies_2.12 | | | &check; | &check; | &check; | &check; | | |
| spark-bigquery-with-dependencies_2.11 | &check; | &check; | | | | | | |

### Maven / Ivy Package Usage
The connector is also available from the
Expand All @@ -107,6 +110,7 @@ repository. It can be used using the `--packages` option or the

| version | Connector Artifact |
|------------|------------------------------------------------------------------------------------|
| Spark 3.4 | `com.google.cloud.spark:spark-3.4-bigquery:${next-release-tag}` |
| Spark 3.3 | `com.google.cloud.spark:spark-3.3-bigquery:${next-release-tag}` |
| Spark 3.2 | `com.google.cloud.spark:spark-3.2-bigquery:${next-release-tag}` |
| Spark 3.1 | `com.google.cloud.spark:spark-3.1-bigquery:${next-release-tag}` |
Expand Down

0 comments on commit d04164a

Please sign in to comment.