Skip to content

Commit

Permalink
[SPARK] GcsVendedTokenProvider
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Sep 7, 2024
1 parent 8a316d4 commit 0def8c2
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 0 deletions.
28 changes: 28 additions & 0 deletions docs/spark-integration/GcsVendedTokenProvider.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# GcsVendedTokenProvider

`GcsVendedTokenProvider` is an `AccessTokenProvider` to provide access tokens for [UCSingleCatalog](UCSingleCatalog.md) to [load tables](UCSingleCatalog.md#loadTable) from [Google Cloud Storage](https://cloud.google.com/storage).

`GcsVendedTokenProvider` can be configured in a Spark application using the following configuration:

```text
spark.hadoop.fs.AbstractFileSystem.gs.impl com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
spark.hadoop.fs.gs.auth.access.token.provider io.unitycatalog.connectors.spark.GcsVendedTokenProvider
spark.hadoop.fs.gs.auth.type ACCESS_TOKEN_PROVIDER
```

## getAccessToken { #getAccessToken }

??? note "AccessTokenProvider"

```java
AccessToken getAccessToken()
```

`getAccessToken` is part of the `AccessTokenProvider` abstraction.

`getAccessToken` creates a new `AccessToken` based on the following Hadoop Configuration properties:

Property | Hadoop Property
-|-
`token` | `fs.gs.auth.access.token.credential`
`expirationTime` | `fs.gs.auth.access.token.expiration`
7 changes: 7 additions & 0 deletions docs/spark-integration/UCSingleCatalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

`UCSingleCatalog` is a `TableCatalog` ([Spark SQL]({{ book.spark_sql }}/connector/catalog/TableCatalog/)).

`UCSingleCatalog` supports [loading tables](#loadTable) from the following cloud object storages:

Cloud Object Storage | Scheme
-|-
[Amazon S3](https://aws.amazon.com/s3/) | `s3://`
[Google Cloud Storage](https://cloud.google.com/storage) | `gs://`

## DeltaCatalog { #deltaCatalog }

`UCSingleCatalog` creates a `DeltaCatalog` ([Delta Lake]({{ book.delta }}/DeltaCatalog)) when requested to [initialize](#initialize).
Expand Down

0 comments on commit 0def8c2

Please sign in to comment.