Add support for conda lock file

[munishchouhan]

depends upon seqeralabs/libseqera#25
the above PR needs 'git revert d088604' before merging

This PR will add the following

1. upload lockfile to bucket
2. download lockfile from bucket
3. link in build page

[pditommaso]

Good start. To tell the truth, still not sure we should go ahead with this approach or just store the lock file in the surreal db like we are doing for the conda env, even tho it there's the possibility to have the same problem as #559

[munishchouhan]

Good start. To tell the truth, still not sure we should go ahead with this approach or just store the lock file in the surreal db like we are doing for the conda env, even tho it there's the possibility to have the same problem as #559

#559 will be solved in surrealdb version 2.0.0

[pditommaso]

I'm bit confused by this comment in the issue

SURREAL_HTTP_MAX_ML_BODY_SIZE (defaults to 4 GiB)
SURREAL_HTTP_MAX_SQL_BODY_SIZE (defaults to 1 MiB)
SURREAL_HTTP_MAX_RPC_BODY_SIZE (defaults to 4 MiB)
SURREAL_HTTP_MAX_KEY_BODY_SIZE (defaults to 16 KiB)
SURREAL_HTTP_MAX_SIGNUP_BODY_SIZE (defaults to 1 KiB)
SURREAL_HTTP_MAX_SIGNIN_BODY_SIZE (defaults to 1 KiB)
SURREAL_HTTP_MAX_IMPORT_BODY_SIZE (defaults to 4 GiB)

It seems suggesting the default sql body size is 1 MB, instead the error we are hitting is much smaller.

[munishchouhan]

It seems suggesting the default sql body size is 1 MB, instead the error we are hitting is much smaller.

we are using /key routes, which is constrained by 16 KiB
SURREAL_HTTP_MAX_KEY_BODY_SIZE (defaults to 16 KiB)

[munishchouhan]

if we use sql to store it then we can bypass this limit

[pditommaso]

This sounds like a plan. please give it a try

[munishchouhan]

This sounds like a plan. please give it a try

ok sure

[munishchouhan]

There is an issue accessing the Conda lock file. The lock file is present in the generated image, not in the buildkit container we are running in Wave.
so either we need to pull the image in another pod and get the file or we need to generate conda lockfile from conda.yml file

cc @ewels @pditommaso

[munishchouhan]

in latter case of generating conda lockfile from conda file, we still need another job to achieve that

[ewels]

Better to get the file from the container - I was trying to avoid generating the lock file separately because then there's no absolute guarantee that it'll end up the same as the actual environment. If it comes from the environment itself it's certain.

[ewels]

Can we print the lock file to stdout and then capture that from the build?

[munishchouhan]

Can we print the lock file to stdout and then capture that from the build?

we can do this:

FROM {{base_image}}
COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
RUN micromamba install -y -n base -f /tmp/conda.yml \
    {{base_packages}}
    && micromamba env export --explicit > environment.lock \
    && cat environment.lock
    && micromamba clean -a -y
RUN 
USER root
ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

I ran it for conda package 'bwa'
i got this in the stdout

#10 12.24 # This file may be used to create an environment using:
#10 12.24 # $ conda create --name <env> --file <this file>
#10 12.24 # platform: linux-aarch64
#10 12.24 @EXPLICIT

[ewels]

Exactly - that works!

Were there a load of lines after the @EXPLICIT yeah?

We'd need to remove the line prefixes but that's all I think..

[pditommaso]

A better approach (maybe) could be: 1) creating the container "locally"; 2) copy the lock file from the built container via buildkit; 3) uploading it to the registry.

Something similar is done for singularity, here.

[munishchouhan]

A better approach (maybe) could be: 1) creating the container "locally"; 2) copy the lock file from the built container via buildkit; 3) uploading it to the registry.

Something similar is done for singularity, here.

Ok sure, I will try this one

[pditommaso]

what if the lock file contains a ' ?

[munishchouhan]

good point
I have not tested yet with surrealDB
I will use bytes datatype to save it

[munishchouhan]

changed to byte[]

[munishchouhan]

Tested locally:

logs:

[munishchouhan]

This PR can only be tested on dev once we release new wave-utils after merging this PR
seqeralabs/libseqera#25

[pditommaso]

Do we have an estimation how big can be a lock file? If it's not too big it could be stored in the db as the conda env file.

[munishchouhan]

not able to find anything on internet,

this is from chatgpt:

1. Conda Lock Files (conda-lock.yml)
Size Range: A typical Conda lock file can range from a few kilobytes to several megabytes, depending on the number of dependencies and platforms included.
Small environment (few dependencies): Can be under 100 KB.
Larger environments (many dependencies, or multi-platform): Can reach several MB (1-5 MB or more).

Real-World Examples:
Conda lock files for a moderately complex data science project (with dependencies like numpy, pandas, tensorflow, etc.) can easily be between 500 KB to 2 MB.
A web application with many dependencies across platforms (npm, pipenv, etc.) might generate a lock file of 2-5 MB.

[pditommaso]

Umm, seems to big. @pinin4fjords do you any clue how big can be - on average - a conda lock file?

[ewels]

• multiqc: 13K
• bwa + samtools: 3.4K
• fastqc: 8.2K
• numpy + pandas: 4.8K
• cuda: 20K

Struggled to get any huge lists of packages to build, so instead pulled out the largest environment.yml files in nf-core/modules:

• spotiflow: build failed
• normaldb: 28K
• intervalfile: 28K
• imc2mc: 15K

They're still not that big, so I went back in time to some DSL1 pipelines, where we had a single conda environment for the entire pipeline:

• rnaseq 1.4.2: 33K
• Sarek 2.7: 63K
• methylseq 1.6.1: 27K

[ewels]

So even with the biggest real-life conda environment that I could find, we're no-where near to the MB range..

[pditommaso]

This is a good point to save the lock directly in the db. it would make it much simpler

[pinin4fjords]

What is the upper limit this would place on the size of the locks? I think we all know that even if locks from nf-core are small, there will be that customer that does something funky to get us into the MB range.

[munishchouhan]

What is the upper limit this would place on the size of the locks? I think we all know that even if locks from nf-core are small, there will be that customer that does something funky to get us into the MB range.

1MB

[pinin4fjords]

What is the upper limit this would place on the size of the locks? I think we all know that even if locks from nf-core are small, there will be that customer that does something funky to get us into the MB range.

1MB

OK, then I'm going to out on a limb and suggest that this would eventually come back to bite us.