Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugins that help to pass credentials for S3 and GCS to remote cluster workers #8883

Closed
dbalabka opened this issue Oct 4, 2024 · 1 comment

Comments

@dbalabka
Copy link

dbalabka commented Oct 4, 2024

I didn't find a simple way to pass credentials to remote workers, such as S3 and GCS, while both are widely used to store data frames.
In this ticket's scope, I propose creating plugins that will help distribute the required keys to remote workers.

GCP credentials
GCP credentials file path is stored in GOOGLE_APPLICATION_CREDENTIALS env variable. The plugin has to create a remote file and pass an env variable with a proper path to workers.

S3 credentials
Like GCP, we must update credential files and store them on each worker.

@dbalabka
Copy link
Author

dbalabka commented Oct 4, 2024

moved to dask/dask-cloudprovider#438

@dbalabka dbalabka closed this as not planned Won't fix, can't repro, duplicate, stale Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants