[Feature] storing config files in local data/output directory instead of in conda env installation directory

[jjtukker]

Is your feature request related to a problem? Please describe.
We are using a cluster (Ubuntu 20.04), with several users that each have their own home directory and log in remotely. We have a central conda environment (on /scratch) which can then be used by different users via a link to their home individual miniconda environments. Unfortunately, this does not work for cellfinder since it saves config files in the installation env, rather than in the home directory (or wherever the data/output files are). This leads to permission errors because the central conda environment should not be changeable by users. Even if we do give permission, the config file refers to the local user directory of whoever first writes it, which creates problems.

Describe the solution you'd like
It would be great if cellfinder could save all files in the output folder instead of in the conda env directory (right now there are two conf files which are saved, in our case, in "/scratch/rootconda/envs/cellfinder/lib/python3.10/site-packages/bg_atlasapi/" and "/scratch/rootconda/envs/cellfinder/lib/python3.10/site-packages/cellfinder_core/config/"). Or, if this might cause problems for existing setups, to add an option on the command-line to say "saveconfiglocal" or something like that.

Describe alternatives you've considered
Unfortunately dockers are not a viable solution on a multi-user system like ours. Generating copies of the central conda environment on local directories does not work (we tried cp, conda clone, and conda-pack but having pip and conda together causes problems). Having each user install cellfinder in their local conda env on their home directory is possible, but so far has been challenging for many users (given the challenges related to tensorflow and GPU-compatibility, and issues related to having to use conda/conda-forge and pip together). We think a cleaner separation of specific config files (it seems there are just 2?) from the general installation would be preferable. Hopefully it is easy to implement?

[adamltyson]

Hi @jjtukker,

This should be possible without any changes to the code (which unfortunately i don't have time for). I've installed cellfinder centrally on an HPC system. What I would do is:

• Download the cellfinder default model to a centrally accessible location using cellfinder_download --install-path /path/to/central/storage.
• Manually edit CONDA_ENV/lib/python3.8/site-packages/bg_atlasapi/bg_config.conf to point to where you want to download the atlases, then download any that may be needed with e.g. brainglobe install -a allen_mouse_50um`)

To get around the fudge for the atlas API, you could install bg-atlasapi from GitHub to take advantage of this PR. I'd also welcome PR to cellfinder based on that other PR to fix this properly.

I think this means that users won't have to edit any files in the conda env, but this isn't how I've set up shared installations before. Previously, I''ve set up a cellfinder module and I could share the set up with you if the sysadmins would allow it.

Adam