Dev (#24)

* Strain from CoupledRec (#16)

* Additional documentation for multipeak phasing

* Update setup.py

* Update meta.yaml

* Refactored get_centered to reduce number of numpy function calls.

* Added a bunch of functions to cplib.py

* Get strain from final reconstruction. This commit also contains commented code that would implement the partial fourier constraint with free-varying pixels according to a preset mask.

* Added several new methods to the cohlib signature:

- diff
- gradient
- argmin
- take_along_axis
- moveaxis
- lstsq
- zeros
- indices
- concatenate

These methods have been implemented in cplib and nplib, but will raise NotImplementedError if called from torchlib or aflib.

* Removed some experimental code that accidentally got included in this version.

---------

Co-authored-by: Barbara Frosik <[email protected]>

* fixed interface issue with cohere-ui after merge

* fixed interface issue after merge

* Delete reconstruction_GA_HPC.py

* moved cleaning memory to lib

* Delete reconstruction_HPC_GA.py

* fixed problrm with parameter

* fixed imoprt name

* fixed docs for developers

* supressed warnings in verifier when parsing device

* fixed crop issue in ui

* Autodata (#18)

* changes for autodata option

* synch with ui

---------

Co-authored-by: bfrosik <[email protected]>

* added debug option

* moved utils from cohere-ui to cohere core and cleane up

* cleaned up utils

* support for auto data for separate scan ranges

* aligned with cohere-ui

* align with cohere-ui

* relaxed verifier for auto data

* relaxed verifier

* aligning with ui

* added missing save_dir in reconstruction scripts

* added for hpc

* sync with cohere-ui module

* removed warning

* added cluster capability

* removed print statements

* cleaned up, bug fixes

* added description for cluster, fixed host naming issue

* fixed typo

* Update .readthedocs.yaml

* Update .readthedocs.yaml

* Update .readthedocs.yaml

* Update .readthedocs.yaml

* Update .readthedocs.yaml

* Update README.md

* Update README.md

* Update README.md

* fixed verifier, aligned cohere-ui

* aligned with cohere-ui module

* formatted text, fixed verifier for multipeak

* blocked auto binning for mp and aligned ui

* return correlation error in addition to aligned array

* aligned ui

* aligned with cohere-ui

* added description in verifier, aligned with ui

* Updated multipeak capabilities (Dev) (#22)

* Additional documentation for multipeak phasing

* Update setup.py

* Update meta.yaml

* Refactored get_centered to reduce number of numpy function calls.

* Added a bunch of functions to cplib.py

* Get strain from final reconstruction. This commit also contains commented code that would implement the partial fourier constraint with free-varying pixels according to a preset mask.

* Added several new methods to the cohlib signature:

- diff
- gradient
- argmin
- take_along_axis
- moveaxis
- lstsq
- zeros
- indices
- concatenate

These methods have been implemented in cplib and nplib, but will raise NotImplementedError if called from torchlib or aflib.

* Removed some experimental code that accidentally got included in this version.

* assigned release tag

* fixed doc for developers

* fixed crop issue in ui

* Added the following functions to cplib.py: amin(), affine_transform(), pad(), histogram2d(), calc_nmi(), calc_ehd().

These have only been implemented in cplib, other libraries are only implemented as stubs.

* Significant changes that should have been committed earlier:

- Resampling is now included in the phasing process.
- Added several new error metrics, including normalized mutual information (NMI) and expected histogram deviation (EHD)
- Apply the support constraint to the full object during ER iterations (to prevent buildup of HIO feedback)
- Fixed incorrect normalization when projecting to each Bragg peak.
- Added a "control_peak" option to exclude one peak from the phasing process to use for unbiased error calculations.
- Added a "calc_strain" option which can skip the slow strain calculation.
- Added a "fast_resample" option to toggle whether the original or resampled data is used.

---------

Co-authored-by: Barbara Frosik <[email protected]>

* aligned with ui

* updated torch lib

* fixed torchlib

* removed unused function, aligned ui

* synch ui

* revised fftconvolve function

* added comments to op flow, and simplified

* modified get_ratio

* removed defaults in cohere-core, added checks for misconfiguration

* added error checks and prints of error messages if fail

* removed code to activate lpf at last iteration

* aligned with cohere-ui

* added tqdm package in setup.py

* added example of different sw type in features

* moved checks based on type

* change definition of algorithms dict to include to_direct and to_reciprocal

* refactored cluster handling

* changed API to receive hostfile instead of arbitrary naming it

* changed api

* Removed Support class and moved support array to Rec

* renamed phase modulus to phase constrain

* change description, slightly modified code

* updated documentation

* corrected docs

* modified for new release

* modified to do beta buid

* refactored to set defaults through get, corrected decs

* modified doc conf.py

* modified doc conf.py

* updated docs

* cleaned up docs

* aligning with cohere-ui

---------

Co-authored-by: Nick Porter <[email protected]>
Co-authored-by: Barbara Frosik <[email protected]>
Co-authored-by: pfrosik <[email protected]>
Co-authored-by: cxduser <[email protected]>

.readthedocs.yaml

@@ -5,24 +5,28 @@
 # Required
 version: 2
 
-# Set the version of Python and other tools you might need
-#build:
-#    python: "3.8"
+# Set the OS, Python version and other tools you might need
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.12"
     # You can also specify other tool versions:
-    # nodejs: "16"
-    # rust: "1.55"
-    # golang: "1.17"
+    # nodejs: "19"
+    # rust: "1.64"
+    # golang: "1.19"
 
-# Build documentation in the docs/ directory with Sphinx
+# Build documentation in the "docs/" directory with Sphinx
 sphinx:
    configuration: docs/source/conf.py
 
-# If using Sphinx, optionally build your docs in additional formats such as PDF
+# Optionally build your docs in additional formats such as PDF and ePub
 # formats:
 #    - pdf
+#    - epub
 
-# Optionally declare the Python requirements required to build your docs
-python:
-   version: 3.8
-   install:
-   - requirements: requirements.txt
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+# python:
+#    install:
+#    - requirements: docs/requirements.txt

README.md

@@ -1,6 +1,6 @@
-[![Documentation Status](https://readthedocs.org/projects/cohere/badge/?version=latest)](http://cohere.readthedocs.io/en/latest/?badge=latest)
+[![Documentation Status](https://readthedocs.org/projects/cohere-dev/badge/?version=latest)](http://cohere-dev.readthedocs.io/en/latest/?badge=latest)
 
-Project home page: https://cohere.readthedocs.io/
+Project home page: [https://cohere-dev.readthedocs.io/](https://cohere-dev.readthedocs.io/)
 
 The cohere package provides tools for reconstruction of image of a nanoscale structures from data obtained using Bragg Coherent Diffraction Imaging technique.
 

build.sh

@@ -1 +1 @@
-python setup.py install
+python setup.py install

cohere_core/controller/AI_guess.py

@@ -51,84 +51,6 @@ def __init__(self, model_file):
                                              outputs=model.get_layer('phi').output)
 
 
-def threshold_by_edge(fp: np.ndarray) -> np.ndarray:
-    # threshold by left edge value
-    mask = np.ones_like(fp, dtype=bool)
-    mask[tuple([slice(1, None)] * fp.ndim)] = 0
-    zero = 1e-6
-    cut = np.max(fp[mask])
-    binary = np.zeros_like(fp)
-    binary[(np.abs(fp) > zero) & (fp > cut)] = 1
-    return binary
-
-
-def select_central_object(fp: np.ndarray) -> np.ndarray:
-    import scipy.ndimage as ndimage
-    zero = 1e-6
-    binary = np.abs(fp)
-    binary[binary > zero] = 1
-    binary[binary <= zero] = 0
-
-    # cluster by connectivity
-    struct = ndimage.morphology.generate_binary_structure(fp.ndim,
-                                                          1).astype("uint8")
-    label, nlabel = ndimage.label(binary, structure=struct)
-
-    # select largest cluster
-    select = np.argmax(np.bincount(np.ravel(label))[1:]) + 1
-
-    binary[label != select] = 0
-
-    fp[binary == 0] = 0
-    return fp
-
-
-def get_central_object_extent(fp: np.ndarray) -> list:
-    fp_cut = threshold_by_edge(np.abs(fp))
-    need = select_central_object(fp_cut)
-
-    # get extend of cluster
-    extent = [np.max(s) + 1 - np.min(s) for s in np.nonzero(need)]
-    return extent
-
-
-def get_oversample_ratio(fp: np.ndarray) -> np.ndarray:
-    """ get oversample ratio
-		fp = diffraction pattern
-	"""
-    # autocorrelation
-    acp = np.fft.fftshift(np.fft.ifftn(np.abs(fp)**2.))
-    aacp = np.abs(acp)
-
-    # get extent
-    blob = get_central_object_extent(aacp)
-
-    # correct for underestimation due to thresholding
-    correction = [0.025, 0.025, 0.0729][:fp.ndim]
-
-    extent = [
-        min(m, s + int(round(f * aacp.shape[i], 1)))
-        for i, (s, f, m) in enumerate(zip(blob, correction, aacp.shape))
-    ]
-
-    # oversample ratio
-    oversample = [
-        2. * s / (e + (1 - s % 2)) for s, e in zip(aacp.shape, extent)
-    ]
-    return np.round(oversample, 3)
-
-
-def Resize(IN, dim):
-    ft = np.fft.fftshift(np.fft.fftn(IN)) / np.prod(IN.shape)
-
-    pad_value = np.array(dim) // 2 - np.array(ft.shape) // 2
-    pad = [[pad_value[0], pad_value[0]], [pad_value[1], pad_value[1]],
-           [pad_value[2], pad_value[2]]]
-    ft_resize = ut.adjust_dimensions(ft, pad)
-    output = np.fft.ifftn(np.fft.ifftshift(ft_resize)) * np.prod(dim)
-    return output
-
-
 def match_oversample_diff(
     diff: np.ndarray,
     fr: Union[list, np.ndarray, None] = None,
@@ -144,7 +66,6 @@ def match_oversample_diff(
     # adjustment needed to match oversample ratio
     change = [np.round(f / t).astype('int32') for f, t in zip(fr, to)]
     change = [np.max([1, c]) for c in change]
-
     diff = ut.binning(diff, change)
     # crop diff to match output shape
     shape_arr = np.array(shape)
@@ -293,7 +214,7 @@ def run_AI(data, model_file, dir):
 
     # prepare data to make the oversampling ratio ~3
     wos = 3.0
-    orig_os = get_oversample_ratio(data)
+    orig_os = ut.get_oversample_ratio(data)
     # match oversampling to wos
     wanted_os = [wos, wos, wos]
     # match diff os
@@ -314,7 +235,7 @@ def run_AI(data, model_file, dir):
     pred_obj = preds_amp * np.exp(1j * preds_phi)
 
     # match object size with the input data
-    pred_obj = Resize(pred_obj, inshape)
+    pred_obj = ut.Resize(pred_obj, inshape)
 
     pad_value = np.array(data.shape) // 2 - np.array(pred_obj.shape) // 2
     pad = [[pad_value[0], pad_value[0]], [pad_value[1], pad_value[1]],
@@ -352,7 +273,7 @@ def start_AI(pars, datafile, dir):
         print ('no AI_trained_model in config')
         return None
     if not os.path.isfile(pars['AI_trained_model']):
-        print('there is no file', pars['AI_trained_model'])
+        print(f'there is no file {pars["AI_trained_model"]}')
         return None
 
     if datafile.endswith('tif') or datafile.endswith('tiff'):
@@ -372,7 +293,7 @@ def start_AI(pars, datafile, dir):
         return None
 
     # The results will be stored in the directory <experiment_dir>/AI_guess
-    ai_dir = dir + '/results_AI'
+    ai_dir = ut.join(dir, 'results_AI')
     if os.path.exists(ai_dir):
         pass
     else:

cohere_core/controller/__init__.py

@@ -1,3 +1,5 @@
+from .mpi_cmd import *
+from .reconstruction_populous_GA import *
 from .reconstruction_GA import *
 from .reconstruction_single import *
 from .reconstruction_multi import *