Packaging workflow files with poetry

[akhanf]

Came across some difficulties trying to get the workflow files (e.g. config/, workflow/, and resources/) properly packaged up when building with poetry/pyproject, specifically when the workflow exists in the root directory of the repository.

In order to have the files included in the build package, they need to be in the package sub-directory. One workaround was to create symlinks in the root directory. I've done this in funcmasker-flex.

Let me know what you guys think about this or if there is another way to do this. The workflow files existing as symlinks is not ideal, and means that the repository doesn't automatically get included in the snakemake workflow catalog

A hacky fix that could allow the workflow files to exist in the main directory (and not be symlinks) could be to move/copy the files to the package directory just before building..

[pvandyken]

Will include do it?

# pyproject.toml
include = ["config/", "workflow/", "resources/"]

[akhanf]

Ya, but then it ends up including them in the site-packages folder when installed, which is weird... there are some open issues related to this on the poetry github

[pvandyken]

So just to make sure I understand, in this scenario, we're assuming the user is not cloning the repository or anything, this is sort of the "Bidsapp" mode, black box code kind of thing and we want to be able to pip install the app and just run it?

I'm wondering if we can do some sort of on-the-fly download when run.py is called. I can't remember the name of that snakemake command that basically just clones and downloads a workflow. But if it has an api, it might be useful. So basically, a simple pipe install wouldn't download any snakemake workflow, that would be handled after. But I don't know if that would actually work because I can't find that snakemake command.

On another note, I was browsing through the #snakemake tag on github, and struggled to find any apps that actually followed the prescribed snakemake protocol! Most either put the workflow in app-name/ or in a dedicated snakemake folder. I'd have to guess it's because of similar issues: most apps were installable through Conda (and surely pip behind the scenes).