improve container credentials retrieval using workflow compute environment #303

munishchouhan · 2023-09-28T11:01:11Z

Added workflowId in PlatformId
Added Models to get workflow information
Added API call in TowerClient to get workflow information using workflowId
Added findComputeCreds to get the credential id from workflow information
Added Unit Test

…l-using-workflow-compute-environment

…s-retrieval-using-workflow-compute-environment' into 223-improve-container-credentials-retrieval-using-workflow-compute-environment

munishchouhan · 2023-09-29T12:01:13Z

@pditommaso how can I test this feature locally?

pditommaso · 2023-09-29T12:27:35Z

run tower in your computer
run wave in your compute
create AWS Batch Compute env having permissions to pull containers from AWS ECR
run nextflow script using a container in that ECR using local tower and wave

munishchouhan · 2023-10-02T11:12:28Z

@pditommaso I have tried using Tower also, but same error
Can you please share, where nextflow sends workflow id to wave?

munishchouhan · 2023-10-02T11:13:42Z

ok i found its there in wave plugin
https://github.com/nextflow-io/nextflow/blob/f0d5cc5c674e4e18424e431a5f50930dcd9906c2/plugins/nf-wave/src/main/io/seqera/wave/plugin/SubmitContainerTokenRequest.groovy#L57

munishchouhan · 2023-10-02T11:50:46Z

@pabloaledo I have used the master branch of nextflow but still got the same error

…l-using-workflow-compute-environment

munishchouhan · 2023-10-10T08:20:33Z

@pditommaso SubmitContainerTokenRequest in Tower doesn't have WorkflowId in it
should i add it there?
https://github.com/seqeralabs/nf-tower-cloud/blob/14a5582fe2aa84b5fda9706e3f6776d9084ced8c/tower-enterprise/src/main/groovy/io/seqera/wave/SubmitContainerTokenRequest.groovy#L32-L79

pditommaso · 2023-10-10T14:01:02Z

it should *not* be the problem. The request is made by Nextflow (that's not used by tower right now)

munishchouhan · 2023-10-11T05:23:12Z

ok i will make the changes in tower

pditommaso · 2023-10-11T06:46:57Z

My bad, I meant "It should not be the problem"

pditommaso · 2023-10-11T06:51:29Z

The flow is like this

Tower adds TOWER_WORKFLOW_ID in the launcher job
The tower client in Nextflow fetchs it and include in the request sent to wave
the rest continues in Wave

Make sure the Wave client find it and submit correctly

munishchouhan · 2023-10-11T10:15:09Z

Found the issue
The parameter name was different in Wave-API and Nextflow
should we use wave-api in nextflow and tower for wave models so that it remains consistent?

Created new PR for Wave-API
seqeralabs/libseqera#6

settings.gradle

…l-using-workflow-compute-environment

marcodelapierre · 2023-11-07T09:39:18Z

Munish: tests are passing locally

@pditommaso you can go ahead and review then

…l-using-workflow-compute-environment

Signed-off-by: Paolo Di Tommaso <[email protected]>

munishchouhan · 2024-07-03T19:15:59Z

@munishchouhan Any chance to fix failing tests at your convenience ?

sure

Signed-off-by: Paolo Di Tommaso <[email protected]>

Signed-off-by: munishchouhan <[email protected]>

…l-using-workflow-compute-environment

Signed-off-by: Paolo Di Tommaso <[email protected]>

…l-using-workflow-compute-environment

Signed-off-by: Paolo Di Tommaso <[email protected]>

…l-using-workflow-compute-environment

Signed-off-by: Paolo Di Tommaso <[email protected]>

…l-using-workflow-compute-environment

munishchouhan · 2024-07-09T13:46:32Z

Tested Blob Transfer: Successful

 % wave -i cr.seqera.io/public/nf-jdk:corretto-17-al2023-jemalloc --wave-endpoint https://wave.stage-seqera.io
wave.stage-seqera.io/wt/XXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
(base) munish.chouhan@Munishs-MacBook-Pro ~ % docker pull wave.stage-seqera.io/wt/XXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
corretto-17-al2023-jemalloc: Pulling from wt/XXXXXX/public/nf-jdk
860904071dc6: Pull complete
82160a56be4d: Pull complete
d2d64551932e: Pull complete
f7d1bc77ad09: Pull complete
87ca65aa7e06: Pull complete
Digest: sha256:3f9cf279c1ad0454244469eb52f955dc41072465b310f238e119cd8cebb6f067
Status: Downloaded newer image for wave.stage-seqera.io/wt/XXXXXX/public/nf-jdk:corretto-17-al2023-jemalloc
wave.stage-seqera.io/wt/XXXXXX/public/nf-jdk:corretto-17-al2023-jemalloc

munishchouhan · 2024-07-09T14:11:25Z

Tested build, scan and build log transfer: successful

% /bin/zsh /Users/munish.chouhan/testing_ground/wave_testing/build-images_stage.sh
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard--26c87e08d44802ba
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:salmon_numpy--31f71aba34cc9f18
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
picard_numpy--e90fce5ae5a4f7c7: Pulling from wt/xxxxxxx/wave/build/stage
6360b3717211: Pull complete 
2ec3f7ad9b3c: Pull complete 
7716ca300600: Pull complete 
4f4fb700ef54: Pull complete 
8c61d418774c: Pull complete 
03dae77ff45c: Pull complete 
aab7f787139d: Pull complete 
837d55536720: Pull complete 
897362c12ca7: Pull complete 
3893cbe24e91: Pull complete 
d1b61e94977b: Pull complete 
57d9b5e475d4: Pull complete 
a4c883d12ac5: Pull complete 
Digest: sha256:371f85c396a177dfa243323cdcdd63273b4589a7b9e5252bd45bbdefdd130470
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview wave.stage-seqera.io/wt/f7d4cfa17e01/wave/build/stage:picard_numpy--e90fce5ae5a4f7c7
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxx/wave/build/stage:picard--26c87e08d44802ba
picard--26c87e08d44802ba: Pulling from wt/xxxxxxx/wave/build/stage
6360b3717211: Already exists 
2ec3f7ad9b3c: Already exists 
7716ca300600: Already exists 
4f4fb700ef54: Already exists 
8c61d418774c: Already exists 
03dae77ff45c: Already exists 
aab7f787139d: Already exists 
837d55536720: Already exists 
897362c12ca7: Already exists 
3893cbe24e91: Already exists 
d1b61e94977b: Already exists 
b630d4f4ff7e: Pull complete 
744b79fad90d: Downloading [=============>                                     ]  184.4MB/674.6MB

munishchouhan · 2024-07-09T14:19:09Z

Test - Container pull: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -i ubuntu --wave-endpoint https://wave.stage-seqera.io
wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
latest: Pulling from wt/xxxxxxxx/library/ubuntu
eed1663d2238: Pull complete 
Digest: sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest
wave.stage-seqera.io/wt/xxxxxxxx/library/ubuntu:latest

Test - Build using dockerfile: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -f Dockerfile --wave-endpoint https://wave.stage-seqera.io --tower-token xxxxxxxx  --tower-endpoint https://api.cloud.stage-seqera.io
wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % docker pull wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
b4347a6d3486b02c: Pulling from wt/xxxxxxxx/wave/build/stage
ec562eabd705: Pull complete 
Digest: sha256:e9119a211bb40231e677fad0f3f60d2c3bb94fcdcb3b752330a908107889cd55
Status: Downloaded newer image for wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c
wave.stage-seqera.io/wt/xxxxxxxx/wave/build/stage:b4347a6d3486b02c

munishchouhan · 2024-07-09T14:35:00Z

Test: build singularity, freeze and user build repository: successful

(base) munish.chouhan@Munishs-MacBook-Pro wave_testing % wave -s --conda-package pandas --freeze --build-repo docker.io/hrma017/dev  --wave-endpoint https://wave.stage-seqera.io --tower-token xxxxxxx  --tower-endpoint https://api.cloud.stage-seqera.io --platform linux/arm64
oras://docker.io/hrma017/dev:pandas--e7eed9f3222a1997
(base) munish.chouhan@Munishs-MacBook-Pro wave_testing %  docker run  --privileged quay.io/singularity/singularity:v3.11.4-slim-arm64 pull oras://docker.io/hrma017/dev:pandas--e7eed9f3222a1997
INFO:    Downloading oras image

pditommaso · 2024-07-09T15:18:08Z

@munishchouhan all green in your side?

VERSION

munishchouhan · 2024-07-09T15:19:23Z

@munishchouhan all green in your side?

testing one last thing. will update here soon

munishchouhan · 2024-07-09T15:24:11Z

test: ran rnasef to create fusion-based images: successful

(base) munish.chouhan@Munishs-MacBook-Pro example-bonus % bash run.sh
Nextflow 24.04.2 is available - Please consider updating your version to it
N E X T F L O W  ~  version 23.10.1
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [55133f624d]
Launching `https://github.com/nextflow-io/rnaseq-nf` [happy_volhard] DSL2 - revision: 88b8ef803a [master]
 R N A S E Q - N F   P I P E L I N E
 ===================================
 transcriptome: /Users/munish.chouhan/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa
 reads        : /Users/munish.chouhan/.nextflow/assets/nextflow-io/rnaseq-nf/data/ggal/ggal_gut_{1,2}.fq
 outdir       : results
 
executor >  local (fusion enabled) (4)
[66/01293a] process > RNASEQ:INDEX (ggal_1_48850000_49020000) [100%] 1 of 1 ✔
[51/3d1445] process > RNASEQ:FASTQC (FASTQC on ggal_gut)      [100%] 1 of 1 ✔
[2a/8846cc] process > RNASEQ:QUANT (ggal_gut)                 [100%] 1 of 1 ✔
[ed/f7a053] process > MULTIQC                                 [100%] 1 of 1 ✔

Done! Open the following report in your browser --> results/multiqc_report.html

Completed at: 09-Jul-2024 17:23:06
Duration    : 2m 10s
CPU hours   : 0.1
Succeeded   : 4

munishchouhan · 2024-07-09T15:25:10Z

@pditommaso All green from my side

Add functionality to fetch ComputeEnv from tower

93bb700

munishchouhan linked an issue Sep 28, 2023 that may be closed by this pull request

Improve container credentials retrieval using workflow compute environment #223

Closed

Merge branch 'master' into 223-improve-container-credentials-retrieva…

618436a

…l-using-workflow-compute-environment

munishchouhan mentioned this pull request Sep 29, 2023

Improve container credentials retrieval using workflow compute environment seqeralabs/libseqera#5

Merged

munishchouhan requested a review from pditommaso September 29, 2023 07:52

munishchouhan added 2 commits September 29, 2023 10:34

updated io.seqera:wave-api:0.6.0

71b5f0c

Merge remote-tracking branch 'origin/223-improve-container-credential…

26500f0

…s-retrieval-using-workflow-compute-environment' into 223-improve-container-credentials-retrieval-using-workflow-compute-environment

debug log added for retrieved workflow info

001c20a

munishchouhan self-assigned this Oct 2, 2023

Merge branch 'master' into 223-improve-container-credentials-retrieva…

402a61c

…l-using-workflow-compute-environment

renamed towerWorkFlowId to workflowId

a991c88

pditommaso reviewed Oct 11, 2023

View reviewed changes

settings.gradle Outdated Show resolved Hide resolved

munishchouhan added 4 commits October 12, 2023 08:27

minor change

4e33c00

minor change

cda3733

Merge branch 'master' into 223-improve-container-credentials-retrieva…

374f844

…l-using-workflow-compute-environment

Merge branch 'master' into 223-improve-container-credentials-retrieva…

b4b9089

…l-using-workflow-compute-environment

munishchouhan added 2 commits November 7, 2023 15:09

Merge branch 'master' into 223-improve-container-credentials-retrieva…

0e9617a

…l-using-workflow-compute-environment

Merge branch 'master' into 223-improve-container-credentials-retrieva…

b30b564

…l-using-workflow-compute-environment

Simplify logic [ci skip]

4a83726

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso and others added 6 commits July 3, 2024 21:50

Add AWS keys mapping [ci ksip]

60b12e7

Signed-off-by: Paolo Di Tommaso <[email protected]>

fixed tests

6c10727

Signed-off-by: munishchouhan <[email protected]>

Merge branch 'master' into 223-improve-container-credentials-retrieva…

cb4c027

…l-using-workflow-compute-environment

Merge branch 'master' into 223-improve-container-credentials-retrieva…

afdf54e

…l-using-workflow-compute-environment

Fix tests

a0f755c

Signed-off-by: Paolo Di Tommaso <[email protected]>

Minor improvement

d11c7b9

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso marked this pull request as ready for review July 7, 2024 17:58

pditommaso and others added 7 commits July 7, 2024 20:00

Minor change

49e0de1

Signed-off-by: Paolo Di Tommaso <[email protected]>

[release] bump version 1.9.0-B1

6ffb091

Signed-off-by: Paolo Di Tommaso <[email protected]>

Merge branch 'master' into 223-improve-container-credentials-retrieva…

25a2723

…l-using-workflow-compute-environment

[release] bump version 1.9.0-B2

75bfd50

Signed-off-by: Paolo Di Tommaso <[email protected]>

Merge branch 'master' into 223-improve-container-credentials-retrieva…

60163c8

…l-using-workflow-compute-environment

[release] bump 1.9.0-B3

bb2a8b1

Signed-off-by: Paolo Di Tommaso <[email protected]>

Merge branch 'master' into 223-improve-container-credentials-retrieva…

d5e3fcf

…l-using-workflow-compute-environment

pditommaso reviewed Jul 9, 2024

View reviewed changes

VERSION Outdated Show resolved Hide resolved

Update VERSION [ci skip]

93f4c65

pditommaso approved these changes Jul 9, 2024

View reviewed changes

pditommaso merged commit ec89522 into master Jul 9, 2024
1 check passed

pditommaso deleted the 223-improve-container-credentials-retrieval-using-workflow-compute-environment branch July 9, 2024 18:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve container credentials retrieval using workflow compute environment #303

improve container credentials retrieval using workflow compute environment #303

munishchouhan commented Sep 28, 2023 •

edited

Loading

munishchouhan commented Sep 29, 2023

pditommaso commented Sep 29, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 10, 2023 •

edited

Loading

pditommaso commented Oct 10, 2023 •

edited

Loading

munishchouhan commented Oct 11, 2023

pditommaso commented Oct 11, 2023

pditommaso commented Oct 11, 2023

munishchouhan commented Oct 11, 2023

marcodelapierre commented Nov 7, 2023 •

edited

Loading

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024 •

edited

Loading

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024

pditommaso commented Jul 9, 2024

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024 •

edited

Loading

munishchouhan commented Jul 9, 2024

improve container credentials retrieval using workflow compute environment #303

improve container credentials retrieval using workflow compute environment #303

Conversation

munishchouhan commented Sep 28, 2023 • edited Loading

munishchouhan commented Sep 29, 2023

pditommaso commented Sep 29, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 2, 2023

munishchouhan commented Oct 10, 2023 • edited Loading

pditommaso commented Oct 10, 2023 • edited Loading

munishchouhan commented Oct 11, 2023

pditommaso commented Oct 11, 2023

pditommaso commented Oct 11, 2023

munishchouhan commented Oct 11, 2023

marcodelapierre commented Nov 7, 2023 • edited Loading

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024 • edited Loading

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024

pditommaso commented Jul 9, 2024

munishchouhan commented Jul 9, 2024

munishchouhan commented Jul 9, 2024 • edited Loading

munishchouhan commented Jul 9, 2024

munishchouhan commented Sep 28, 2023 •

edited

Loading

munishchouhan commented Oct 10, 2023 •

edited

Loading

pditommaso commented Oct 10, 2023 •

edited

Loading

marcodelapierre commented Nov 7, 2023 •

edited

Loading

munishchouhan commented Jul 9, 2024 •

edited

Loading

munishchouhan commented Jul 9, 2024 •

edited

Loading