Skip to content

Commit

Permalink
AWS S3 Configuration
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Oct 20, 2024
1 parent 725059c commit 6bec491
Show file tree
Hide file tree
Showing 5 changed files with 96 additions and 5 deletions.
60 changes: 59 additions & 1 deletion docs/credential-vending/AwsCredentialVendor.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,61 @@
# AwsCredentialVendor

`AwsCredentialVendor` is...FIXME
`AwsCredentialVendor` is used to [vend S3 bucket credentials](#vendAwsCredentials) for [CredentialOperations](CredentialOperations.md#awsCredentialVendor) (to [vend credentials](CredentialOperations.md#vendCredential) for `s3://` storage scheme).

`AwsCredentialVendor` uses [server.properties](../server/ServerPropertiesUtils.md#getS3Configurations) configuration file for S3 bucket security configurations.

!!! note "AWS Security Token Service (STS)"
`AwsCredentialVendor` uses [AWS Security Token Service (STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) to request [temporary, limited-privilege security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html):

* An access key ID
* A secret access key
* A security (or session) token

## S3 Configurations { #s3Configurations }

```java
Map<String, S3StorageConfig> s3Configurations
```

`AwsCredentialVendor` initializes `s3Configurations` based on [server.properties](../server/ServerPropertiesUtils.md#getS3Configurations) configuration file.

This `s3Configurations` is used to look up [S3StorageConfig](S3StorageConfig.md)s to [vend S3 bucket credentials](#vendAwsCredentials).

## Vend Credentials { #vendAwsCredentials }

```java
Credentials vendAwsCredentials(
CredentialContext context)
```

`vendAwsCredentials` looks up the [S3 bucket](CredentialContext.md#getStorageBase) (of the given [CredentialContext](CredentialContext.md)) in the [S3 Configurations](#s3Configurations).

??? note "BaseException"
`vendAwsCredentials` reports a `BaseException` when the given storage base could not be found:

```text
S3 bucket configuration not found.
```

If a [session token](S3StorageConfig.md#getSessionToken) is defined, `vendAwsCredentials` returns a "static session" `Credentials` with the following:

* [Access Key](S3StorageConfig.md#getAccessKey)
* [Secret Key](S3StorageConfig.md#getSecretKey)
* [Session Token](S3StorageConfig.md#getSessionToken)

Otherwise (with no [session token](S3StorageConfig.md#getSessionToken) defined), `vendAwsCredentials` [gets an AWS STS client](#getStsClientForStorageConfig) to assume a role with the following:

Property | Value
-|-
**Amazon Resource Name (ARN)** of the role to assume | [Role ARN](S3StorageConfig.md#getAwsRoleArn)
**IAM policy** | [IAM policy](AwsPolicyGenerator.md#generatePolicy) for the [privileges](CredentialContext.md#getPrivileges) and the [locations](CredentialContext.md#getLocations)
**Assumed Role Session** | `uc-[randomUUID]`
Duration | 1 hour

In the end, `vendAwsCredentials` requests the `StsClient` for the temporary security credentials (an access key ID, a secret access key, and a security (or session) token).

---

`vendAwsCredentials` is used when:

* `CredentialOperations` is requested to [vend AWS credentials](CredentialOperations.md#vendAwsCredential)
3 changes: 3 additions & 0 deletions docs/credential-vending/AwsPolicyGenerator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# AwsPolicyGenerator

`AwsPolicyGenerator` is...FIXME
11 changes: 11 additions & 0 deletions docs/credential-vending/CredentialContext.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# CredentialContext

## Creating Instance

`CredentialContext` takes the following to be created:

* <span id="storageScheme"> Storage Scheme
* <span id="storageBase"> Storage Base (e.g., S3 bucket)
* <span id="privileges"> Privileges
* <span id="locations"> Locations

`CredentialContext` is created using [CredentialContext.create](#create) factory method.

## Create CredentialContext { #create }

``` java
Expand Down
17 changes: 17 additions & 0 deletions docs/credential-vending/CredentialOperations.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,20 @@ Storage Scheme | Credential Vending Mechanism | TemporaryCredentials
* `TemporaryPathCredentialsService` is requested to [generateTemporaryPathCredential](../server/TemporaryPathCredentialsService.md#generateTemporaryPathCredential)
* `TemporaryTableCredentialsService` is requested to [generateTemporaryTableCredential](../server/TemporaryTableCredentialsService.md#generateTemporaryTableCredential)
* `TemporaryVolumeCredentialsService` is requested to [generateTemporaryTableCredential](../server/TemporaryVolumeCredentialsService.md#generateTemporaryTableCredential)

## vendAwsCredential { #vendAwsCredential }

```java
Credentials vendAwsCredential(
CredentialContext context)
```

`vendAwsCredential` requests the [AwsCredentialVendor](#awsCredentialVendor) to [vendAwsCredentials](AwsCredentialVendor.md#vendAwsCredentials) for the given [CredentialContext](CredentialContext.md).

---

`vendAwsCredential` is used when:

* `CredentialOperations` is requested to [vend credentials](#vendCredential) for`s3://` storage scheme
* `FileIOFactory` is requested to [getAwsCredentialsProvider](../iceberg/FileIOFactory.md#getAwsCredentialsProvider)
* `TableConfigService` is requested to [getS3Config](../iceberg/TableConfigService.md#getS3Config)
10 changes: 6 additions & 4 deletions docs/server/ServerPropertiesUtils.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ This `ServerPropertiesUtils` instance is used when:

`ServerPropertiesUtils` [loads server configuration](#loadProperties) from `etc/conf/server.properties` when [created](#creating-instance).

When there is no `etc/conf/server.properties`, the server properties are empty.
When there is no `etc/conf/server.properties`, the server properties is empty.

`ServerPropertiesUtils` uses the properties for the following:

Expand All @@ -47,7 +47,9 @@ When there is no `etc/conf/server.properties`, the server properties are empty.
* [getGcsConfigurations](#getGcsConfigurations)
* [getAdlsConfigurations](#getAdlsConfigurations)

## getGcsConfigurations { #getGcsConfigurations }
## Get Storage Configurations

### GCS { #getGcsConfigurations }

``` java
Map<String, String> getGcsConfigurations()
Expand All @@ -65,13 +67,13 @@ Map<String, String> getGcsConfigurations()

* `GcpCredentialVendor` is [created](../credential-vending/GcpCredentialVendor.md#gcsConfigurations)

## getS3Configurations { #getS3Configurations }
### S3 { #getS3Configurations }

``` java
Map<String, S3StorageConfig> getS3Configurations()
```

`getS3Configurations` creates a `S3StorageConfig` for every collection of `n`-indexed properties from the [server.properties](#properties) file (with `n` starting from `0`):
`getS3Configurations` creates a [S3StorageConfig](../credential-vending/S3StorageConfig.md) for every collection of `n`-indexed properties from the [server.properties](#properties) file (with `n` starting from `0`):

* `s3.bucketPath.[n]`
* `s3.region.[n]`
Expand Down

0 comments on commit 6bec491

Please sign in to comment.