Skip to content

Commit

Permalink
Fix binary deps (#12)
Browse files Browse the repository at this point in the history
* remove binary builds

* remove post-link from build

* pin zlibz

* fix typo

* try changing to py 3.9?

* try changing to py 3.9?

* try changing to py 3.9?

* put statphysbio first

* channel ordering
  • Loading branch information
william-galvin authored Oct 28, 2024
1 parent 7d7ee74 commit c473bc5
Show file tree
Hide file tree
Showing 6 changed files with 24 additions and 74 deletions.
7 changes: 3 additions & 4 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@ jobs:

- name: Install Zernikegrams
run: |
conda install -y python=3.10
conda install -y python=3.10 zlib=1.2.13 libzlib=1.2.13 -c conda-forge
conda update conda
conda install -y conda-build
PREFIX=$HOME/local bash devtools/conda-build/post-link.sh
conda build devtools/conda-build --no-test --output-folder ./build -c conda-forge
conda install zernikegrams -c ./build -c conda-forge
conda build devtools/conda-build --no-test --output-folder ./build -c statphysbio -c conda-forge -c bioconda
conda install zernikegrams -c ./build -c statphysbio -c conda-forge -c bioconda
- name: reinstall h5py
run: |
Expand Down
26 changes: 7 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@
```
conda install zernikegrams -c statphysbio -c conda-forge
```
This installs zernikegrams (our package) and it's dependencies in an existing conda environment.

This installs zernikegrams (our package) and its dependencies in an existing conda environment.
Common issues:
- *dssp requires libzlib >=1.3.1 but everything else requires libzlib <1.3* Fix: make sure to install with -c statphysbio before -c bioconda. statphysbio specifically builds a version of dssp for this reason! (But bio conda )

### Requirements
Zernikegrams is distributed through the anaconda package manager, which provides most dependencies in most cases. Notable exceptions include:
- `foldcomp`, which is optional and only necessary if using `--foldcomp` with `structural-info`. If you are, you probably already have it installed, but you can install it with `pip install foldcomp` if not.
- A modern GCC compiler. Most machines have one already, but if you see cryptic messages referencing "GLIBCXX_3.4.30 not found", try `conda install libstdcxx-ng -conda-forge` (if on Linux).
- `argparse`, which comes with almost all Python distributions, but (apparently) not all and is (apparently) not installable with conda. Try `pip install argparse`.

### Supported Platforms
Expand Down Expand Up @@ -49,22 +50,13 @@ conda install zernikegrams -c ./build -c conda-forge --only-deps
```
This installs the dependencies, but not zernikegrams itself. If the dependencies haven't changed since
the latest release, you can also use `conda install zernikegrams -c statphysbio -c conda-forge --only-deps`

4. Build Reduce, the program for adding hydrogens. The easiest way to do this is with
```
PREFIX=$HOME/local bash devtools/conda-build/post-link.sh
```
Which installs the reduce executable in $HOME/local/bin, where structural_info_core expects it. $PREFIX can be something
other than $HOME/local, as long as it's not the current working directory. This will not change where reduce is installed, just where some
temporary files live.

5. Install zernikegrams
4. Install zernikegrams
```
pip install -e . -vv
```
`-e` is for editable mode (changes take effect without reinstalling) and `-vv` is very verbose. Either can be changed.

6. Run `pytest` from the root directory--if everything passes, you're good to go!
1. Run `pytest` from the root directory--if everything passes, you're good to go!

### Testing
Tests should be run with `pytest` from the root directory. It is polite to include new tests with new code (if it can be reasonably tested) and to ensure that new code doesn't break old tests. Bug fixes should include at least one test that fails without the fix and passes with it.
Expand All @@ -73,7 +65,7 @@ Tests should be run with `pytest` from the root directory. It is polite to inclu
If you would like your changes to be reflected in the latest version of the package that's pulled from conda, create a new release. In GitHub, find the "Releases" section (on the right) and follow the instructions. It's polite to version your release as [MAJOR.MINOR.PATCH](https://semver.org/) and provide a detailed description of updates. When you create a new version, a GitHub Actions script will run and (if successful) automatically update the Anaconda repository.

### GitHub Actions
On every push to every branch, `.github/workflows/run-tests.yml` builds the repo from scratch using conda, configures the environment, installs DSSP, and runs `pytest`. If it passes, a green check mark shows up. Currently, we only test using python 3.9 on a Linux x86-64 machine. In the future, we might want to test on more Python versions and macos-13 (x86) and macos-latest (arm)--probably only on pushes to main--using `matrix`.
On every push to every branch, `.github/workflows/run-tests.yml` builds the repo from scratch using conda, configures the environment, and runs `pytest`. If it passes, a green check mark shows up. Currently, we only test using python 3.9 on a Linux x86-64 machine. In the future, we might want to test on more Python versions and macos-13 (x86) and macos-latest (arm)--probably only on pushes to main--using `matrix`.

On every release, `.github/workflows/build_and_upload_conda.yaml` runs, which updates the Anaconda repository. It was configured using [this tutorial](https://github.com/marketplace/actions/build-and-upload-conda-packages). In `devtools/` there is the `meta.yaml` file that conda uses to build the package.

Expand All @@ -92,10 +84,6 @@ As a last resort, we could put `pip install ...` in `post-link.sh`. This is cons
### Building Dependencies from Source
Most Anaconda packages distribute compiled binaries, not source code. Due to limitations of GH Actions runners (e.g., no Linux arm support), it's problematic to rely on GH Actions to compile and distribute code that we need to build from source.

Currently, the only example of this is Reduce. Our approach is to put the build-from-source code in `devtools/conda-build/post-link.sh`, which conda automatically runs on every local machine when zernikegrams is installed.

In general: all of the binaries we compile should be installed in the same, **non-root**, location. Currently, `$HOME/local` seems reasonable. If the code we're compiling uses `cmake`, then `-DCMAKE_INSTALL_PREFIX=$HOME/local` should do that.

### Roadmap
- Support MMFT files as well as .pdb files. Currently there is support for `foldcomp`, but not MMFT.
- Support for other radial basis functions (e.g., bessel)
Expand Down
10 changes: 8 additions & 2 deletions devtools/conda-build/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ requirements:
run:
- python >=3.9
- biopython
- cmake >=3.26 # pinned for DSSP
- cxx-compiler
- dssp
- git
- h5py
- hdf5plugin
Expand All @@ -34,12 +33,14 @@ requirements:
- pyopencl
- pytest
- pytorch
- reduce
- rich
- scikit-learn
- sqlitedict
- stopit
- pyyaml
- zlib <1.3 # pinned for pyrosetta
- libzlib <1.3 # pinned for pyrosetta

test:
imports:
Expand All @@ -56,5 +57,10 @@ about:
license: MIT

extra:
channels:
- statphysbio
- conda-forge
- bioconda
- defaults
recipe-maintainers:
- william-galvin
35 changes: 0 additions & 35 deletions devtools/conda-build/post-link.sh

This file was deleted.

6 changes: 4 additions & 2 deletions zernikegrams/structural_info/get_structural_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
logger = logging.getLogger(__name__)


def parse_args():
def get_parser():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
Expand Down Expand Up @@ -141,8 +141,10 @@ def parse_args():
type=str,
help="[Optional] Directory to save fixed pdb files, if -F or -H is selected"
)
return parser

args = parser.parse_args()
def parse_args():
args = get_parser().parse_args()
if args.pdb_dir is None and args.foldcomp is None:
msg = "pdb_dir or foldcomp must be set"
logger.exception(msg)
Expand Down
14 changes: 2 additions & 12 deletions zernikegrams/structural_info/structural_info_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,9 @@

logger = logging.getLogger(__name__)

REDUCER = os.path.join(
os.path.expanduser("~"),
"local",
"bin",
"reduce"
)
REDUCER = "reduce"

DSSP = os.path.join(
os.path.expanduser("~"),
"local",
"bin",
"mkdssp"
)
DSSP = "mkdssp"
DSSP_VERSION = (
subprocess.run(
f"{DSSP} --version".split(), capture_output=True
Expand Down

0 comments on commit c473bc5

Please sign in to comment.