Merge pull request #366 from ml-struct-bio/v3.3.0-alpha

Version 3.3.0: direct traversal, TestPyPI

.github/workflows/beta_release.yml

@@ -0,0 +1,36 @@
+name: Beta Release
+
+on:
+  push:
+    tags:
+      - '[0-9]+.[0-9]+.[0-9]+*'
+
+jobs:
+  beta-release:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Upgrade setuptools/build
+        run: |
+          pip3 install wheel --upgrade
+          pip3 install setuptools --upgrade
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.9
+
+      - name: Release to TestPyPI
+        if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+
+        run: |
+          python -m pip install --upgrade build
+          python -m build .
+          python -m pip install --upgrade twine
+          twine upload --repository testpypi dist/*

.github/workflows/docs.yml

@@ -2,29 +2,26 @@ name: Docs
 
 on:
   push:
-    branches: [ master, develop, docs ]
-    tags:
-      - '*'
+    branches: [ main ]
 
 jobs:
 
   build_docs:
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python: [3.9]
 
     steps:
       - name: Setup Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
-          python-version: ${{ matrix.python }}
+          python-version: 3.9
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
 
       - name: Install Python dependencies
         run: |
-          pip install sphinx myst-parser sphinx-rtd-theme
-
-      - uses: actions/checkout@v2
+          python -m pip install --upgrade pip
+          pip install sphinx myst-parser sphinx-rtd-theme /home/runner/work/cryodrgn/cryodrgn/
 
       - name: Setup Envvars
         run: |

.github/workflows/main.yml

@@ -1,46 +1,42 @@
-name: CI
+name: Continuous integration testing
 
 on:
   push:
-    branches: [ master, develop ]
+    branches: [ main, develop ]
   pull_request:
-    branches: [ master, develop, vb/imagesoure ]
+    branches: [ main, develop ]
 
 jobs:
   run_tests:
 
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        python: ['3.7', '3.8', '3.9', '3.10']
+        os: [ubuntu-latest, macos-latest]
+        python: ['3.9', '3.10']
       fail-fast: false
 
     steps:
-      - uses: actions/checkout@v2
-
-      - name: Upgrade setuptools
-        run: |
-          pip3 install wheel --upgrade
-          pip3 install setuptools --upgrade
+      - uses: actions/checkout@v4
 
       - name: Setup Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python }}
 
       - name: Install cryoDRGN with dev dependencies
         run: |
-          python -m pip install .[dev]
+          python3 -m pip install .[dev]
 
       - name: Pre-commit checks
         run: |
-          pre-commit run --all-files
+          pre-commit run --all-files --show-diff-on-failure
 
       - name: Pyright
         run: |
           pyright --version
-          pyright
+          #pyright
 
       - name: Pytest
         run: |
-          pytest
+          pytest -v

.github/workflows/release.yml

@@ -2,25 +2,25 @@ name: Release
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
     tags:
-      - '*'
+        - '[0-9]+.[0-9]+.[0-9]+*'
 
 jobs:
   release:
 
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - name: Upgrade setuptools/build
         run: |
           pip3 install wheel --upgrade
           pip3 install setuptools --upgrade
 
       - name: Setup Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: 3.9
 

.gitignore

@@ -12,6 +12,7 @@ __pycache__/
 env/
 build/
 develop-eggs/
+docs/generated/
 dist/
 downloads/
 eggs/

README.md

@@ -5,13 +5,22 @@
 CryoDRGN is a neural network based algorithm for heterogeneous cryo-EM reconstruction. In particular, the method models a *continuous* distribution over 3D structures by using a neural network based representation for the volume.
 
 
-## Documentation:
+## Documentation
 
 The latest documentation for cryoDRGN is available in our [user guide](https://ez-lab.gitbook.io/cryodrgn/), including an overview and walkthrough of cryoDRGN installation, training and analysis. A brief quick start is provided below.
 
 For any feedback, questions, or bugs, please file a Github issue or start a Github discussion.
 
 
+### New in Version 3.3.x
+
+* [NEW] `cryodrgn direct_traversal` to generate interpolations in the conformation latnet space between two points
+* improved interface for `graph` and `pc` traversal methods
+* adding `--datadir` to `cryodrgn abinit_homo` for use with .star files
+* fixing various bugs in Jupyter demonstration notebooks
+* support for TestPyPI beta releases via `pip`
+
+
 ### New in Version 3.2.x
 * [NEW] `cryodrgn_utils clean` for removing extraneous output files from completed experiments
 * [NEW] `cryodrgn_utils fsc`, `cryodrgn_utils fsc_plot`, `cryodrgn_utils gen_mask` adapted from existing scripts;
@@ -182,6 +191,9 @@ The official version 1.0 release. This version introduces several new tools for
     # install cryodrgn
     (cryodrgn) $ pip install cryodrgn
 
+You can alternatively install a newer, less stable, development version of `cryodrgn` using our beta release channel:
+    `(cryodrgn) $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ 'cryodrgn==3.3.0a1.dev4'
+
 More installation instructions are found in the [documentation](https://ez-lab.gitbook.io/cryodrgn/installation).
 
 ## Quickstart: heterogeneous reconstruction with consensus poses
@@ -673,4 +685,4 @@ bioRxiv 2023, https://www.biorxiv.org/content/10.1101/2023.08.18.553799v1
 
 ## Contact
 
-Please submit any bug reports, feature requests, or general usage feedback as a github issue or discussion. 
+Please submit any bug reports, feature requests, or general usage feedback as a github issue or discussion.

cryodrgn/analysis.py

@@ -554,19 +554,24 @@ def selection_fn(trace, points, selector):
 def plot_projections(imgs, labels=None, max_imgs=25):
     if len(imgs) > max_imgs:
         imgs = imgs[:max_imgs]
+
     N = len(imgs)
     nrows = int(np.floor(N**0.5))
     ncols = int(np.ceil(N**0.5))
+
     fig, axes = plt.subplots(
         nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows)
     )
     axes = axes.ravel()
-    for i in range(N):
+
+    for i in range(nrows * ncols):
         axes[i].imshow(imgs[i], cmap="Greys_r")
         if labels is not None:
             axes[i].set_title(labels[i])
+
     for i in range(nrows * ncols):
         axes[i].axis("off")
+
     plt.tight_layout()
     return fig, axes
 

cryodrgn/commands/abinit_homo.py

@@ -94,6 +94,11 @@ def add_args(parser):
     parser.add_argument(
         "--ind", type=os.path.abspath, help="Filter particle stack by these indices"
     )
+    parser.add_argument(
+        "--datadir",
+        type=os.path.abspath,
+        help="Path prefix to particle stack if loading relative paths from a .star or .cs file",
+    )
     parser.add_argument(
         "--lazy",
         action="store_true",
@@ -486,6 +491,7 @@ def save_config(args, dataset, lattice, out_config):
         ind=args.ind,
         window=args.window,
         window_r=args.window_r,
+        datadir=args.datadir,
         ctf=args.ctf,
     )
     if args.tilt is not None:
@@ -553,6 +559,7 @@ def main(args):
         invert_data=args.invert_data,
         ind=ind,
         window=args.window,
+        datadir=args.datadir,
         window_r=args.window_r,
     )
 

cryodrgn/commands/analyze.py

@@ -8,6 +8,7 @@
 import shutil
 from datetime import datetime as dt
 import logging
+import nbformat
 import matplotlib
 import matplotlib.pyplot as plt
 import numpy as np
@@ -386,7 +387,7 @@ def main(args):
             ctf_params = utils.load_pkl(configs["dataset_args"]["ctf"])
             orig_apixs = set(ctf_params[:, 1])
 
-	    # TODO: add support for multiple optics groups
+            # TODO: add support for multiple optics groups
             if len(orig_apixs) > 1:
                 use_apix = 1.0
                 logger.info(
@@ -451,46 +452,35 @@ def main(args):
             num_ksamples=args.ksample,
         )
 
-    # copy over template if file doesn't exist
+    # create demonstration Jupyter notebooks from templates if they don't already exist
     cfg = config.load(cfg)
+    ipynbs = ["cryoDRGN_figures"]
     if cfg["model_args"]["encode_mode"] == "tilt":
-        out_ipynb = f"{outdir}/cryoDRGN_ET_viz.ipynb"
-        if not os.path.exists(out_ipynb):
-            logger.info("Creating jupyter notebook...")
-            ipynb = f"{cryodrgn._ROOT}/templates/cryoDRGN_ET_viz_template.ipynb"
-            shutil.copyfile(ipynb, out_ipynb)
-        else:
-            logger.info(f"{out_ipynb} already exists. Skipping")
-        logger.info(out_ipynb)
+        ipynbs += ["cryoDRGN_ET_viz"]
     else:
-        out_ipynb = f"{outdir}/cryoDRGN_viz.ipynb"
-        if not os.path.exists(out_ipynb):
-            logger.info("Creating jupyter notebook...")
-            ipynb = f"{cryodrgn._ROOT}/templates/cryoDRGN_viz_template.ipynb"
-            shutil.copyfile(ipynb, out_ipynb)
-        else:
-            logger.info(f"{out_ipynb} already exists. Skipping")
-        logger.info(out_ipynb)
-
-        # copy over template if file doesn't exist
-        out_ipynb = f"{outdir}/cryoDRGN_filtering.ipynb"
-        if not os.path.exists(out_ipynb):
-            logger.info("Creating jupyter notebook...")
-            ipynb = f"{cryodrgn._ROOT}/templates/cryoDRGN_filtering_template.ipynb"
-            shutil.copyfile(ipynb, out_ipynb)
+        ipynbs += ["cryoDRGN_viz", "cryoDRGN_filtering"]
+
+    for ipynb in ipynbs:
+        nb_outfile = os.path.join(outdir, f"{ipynb}.ipynb")
+
+        if not os.path.exists(nb_outfile):
+            logger.info(f"Creating demo Jupyter notebook {nb_outfile}...")
+            nb_infile = os.path.join(
+                cryodrgn._ROOT, "templates", f"{ipynb}_template.ipynb"
+            )
+            shutil.copyfile(nb_infile, nb_outfile)
         else:
-            logger.info(f"{out_ipynb} already exists. Skipping")
-        logger.info(out_ipynb)
-
-    # copy over template if file doesn't exist
-    out_ipynb = f"{outdir}/cryoDRGN_figures.ipynb"
-    if not os.path.exists(out_ipynb):
-        logger.info("Creating jupyter notebook...")
-        ipynb = f"{cryodrgn._ROOT}/templates/cryoDRGN_figures_template.ipynb"
-        shutil.copyfile(ipynb, out_ipynb)
-    else:
-        logger.info(f"{out_ipynb} already exists. Skipping")
-    logger.info(out_ipynb)
+            logger.info(f"{nb_outfile} already exists. Skipping")
+
+        # lazily look at the beginning of the notebook for the epoch number to update
+        with open(nb_outfile, "r") as f:
+            filter_ntbook = nbformat.read(f, as_version=nbformat.NO_CONVERT)
+        for i in range(5):
+            filter_ntbook["cells"][i]["source"] = filter_ntbook["cells"][i][
+                "source"
+            ].replace("EPOCH = None", f"EPOCH = {epoch}")
+        with open(nb_outfile, "w") as f:
+            nbformat.write(filter_ntbook, f)
 
     logger.info(f"Finished in {dt.now() - t1}")