Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using AWS secrets in includeConfig doesn't work #5312

Open
bentsherman opened this issue Sep 18, 2024 · 4 comments
Open

Using AWS secrets in includeConfig doesn't work #5312

bentsherman opened this issue Sep 18, 2024 · 4 comments

Comments

@bentsherman
Copy link
Member

We added the ability to use secrets in the includeConfig path in #4840 . This works for local secrets, but not AWS secrets, because AWS secrets requires the xpack-amzn plugin and the config is resolved before plugins are loaded, since the config may include additional plugins.

To fix this I think we would need to load plugins in two phases, adding an "early" phase which happens before the config is resolved.

Not sure how much demand there is for this use case, but it would include any AWS Batch run in Seqera Platform that loads a private config using a secret credential.

cc @markpanganiban

@bentsherman bentsherman changed the title Using secrets in includeConfig doesn't work with AWS secrets Using AWS secrets in includeConfig doesn't work Sep 18, 2024
@pditommaso
Copy link
Member

Bit ugly, because plugins depends on the config 🤦

@pditommaso
Copy link
Member

Likely assuming the AWS config does not rely in the config setting it's possible to load the AWS plugin to resolve the secrets

@pditommaso
Copy link
Member

There are two possible approaches:

  1. Fetching the branch to be used via the provider API, likely here
  2. or using git remote via jgit (see here)

Tagging @tom-seqera for visibility

@ewels
Copy link
Member

ewels commented Nov 19, 2024

Bonus functionality that could be unlocked if we can get AWS credentials in for config loading: the ability to load a Nextflow config file from s3:// (in addition to local / http etc).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants