What would be needed to use Azure Blob Storage? #50
Labels
enhancement
New feature or request
Comments
|
Azure Blob Storage is definitely on our roadmap in very near future (~ in 1-2 months). We use object_store crate, which supports Azure Blob Storage. |
aykut-bozkurt
added a commit
that referenced
this issue
Oct 23, 2024
Only supports Azure Blob uris in the form of `https://{account}.blob.core.windows.net/{container}/key`.
Azure Blob client can be configured with environment variables `AZURE_STORAGE_ACCOUNT_NAME` or `AZURE_STORAGE_SAS_TOKEN`.
Additionally, PR supports following S3 uri forms:
- `s3(a)://{bucket}/key`
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`
Closes #50
aykut-bozkurt
added a commit
that referenced
this issue
Oct 23, 2024
Only supports Azure Blob uris in the form of `https://{account}.blob.core.windows.net/{container}/key`.
Azure Blob client can be configured with environment variables `AZURE_STORAGE_ACCOUNT_NAME` or `AZURE_STORAGE_SAS_TOKEN`.
Additionally, PR supports following S3 uri forms:
- `s3(a)://{bucket}/key`
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`
Closes #50
aykut-bozkurt
added a commit
that referenced
this issue
Oct 24, 2024
Only supports Azure Blob uris in the form of `https://{account}.blob.core.windows.net/{container}/key`.
Azure Blob client can be configured with environment variables `AZURE_STORAGE_ACCOUNT_NAME` or `AZURE_STORAGE_SAS_TOKEN`.
Additionally, PR supports following S3 uri forms:
- `s3(a)://{bucket}/key`
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`
Closes #50
aykut-bozkurt
added a commit
that referenced
this issue
Oct 26, 2024
Supports following Azure Blob uri forms:
- `az://{container}/key`
- `azure://{container}/key`
- `https://{account}.blob.core.windows.net/{container}/key`
**Configuration**
The simplest way to configure object storage is by creating the standard [`~/.azure/config`](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration?view=azure-cli-latest) file:
```bash
$ cat ~/.azure/config
[storage]
account = devstoreaccount1
key = Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
Alternatively, you can use the following environment variables when starting postgres to configure the Azure Blob Storage client:
- `AZURE_STORAGE_ACCOUNT`: the storage account name of the Azure Blob
- `AZURE_STORAGE_KEY`: the storage key of the Azure Blob
- `AZURE_STORAGE_SAS_TOKEN`: the storage SAS token for the Azure Blob
- `AZURE_CONFIG_FILE`: an alternative location for the config file
**Bonus**
Additionally, PR supports following S3 uri forms:
- `s3://{bucket}/key`
- `s3a://{bucket}/key`
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`
Closes #50
aykut-bozkurt
added a commit
that referenced
this issue
Oct 26, 2024
Supports following Azure Blob uri forms:
- `az://{container}/key`
- `azure://{container}/key`
- `https://{account}.blob.core.windows.net/{container}/key`
**Configuration**
The simplest way to configure object storage is by creating the standard [`~/.azure/config`](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration?view=azure-cli-latest) file:
```bash
$ cat ~/.azure/config
[storage]
account = devstoreaccount1
key = Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```
Alternatively, you can use the following environment variables when starting postgres to configure the Azure Blob Storage client:
- `AZURE_STORAGE_ACCOUNT`: the storage account name of the Azure Blob
- `AZURE_STORAGE_KEY`: the storage key of the Azure Blob
- `AZURE_STORAGE_SAS_TOKEN`: the storage SAS token for the Azure Blob
- `AZURE_CONFIG_FILE`: an alternative location for the config file
**Bonus**
Additionally, PR supports following S3 uri forms:
- `s3://{bucket}/key`
- `s3a://{bucket}/key`
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`
Closes #50
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What would be needed to be able to read Parquet files from Azure Blob Storage instead of Amazon S3?
I and some other people read the announcement of this library and for the benefit of everyone thinking I thought I could record the question here. I took a look at the codebase and libraries, but couldn't determine where changes would be needed if someone were to think this too.
Very welcome extension!
The text was updated successfully, but these errors were encountered: