Code and GUI improvements, NoRMCorre and additional file formats support, etc.

[bahanonu]

Main features and changes:

• Increment to CIAtah version 4.5.7.
• NoRMCorre is now supported within main CIAtah GUI and as a switch within turboregMovie (to keep support for surrounding motion correction processing code). This gives users another option for motion correction with non-rigid registration is needed.
  • Note this runs a custom forked version of NoRMCorre in which I moved all code within a normcorre package and re-wrote to make compatible to improve namespace and integration: https://github.com/bahanonu/NoRMCorre.
• Improved support for Bio-Formats files (better file handling, ability to grab data via internal series name, etc.). Additional integration into main CIAtah GUI in next release.
• The main module for pre-processing (modelPreprocessMovieFunction) has several new options, speed/code improvements, NoRMCorre integration, etc. Further support for read-from-disk pre-processing in next release.
• Added support for SlideBook (.sld) files.
• Support for MATLAB's new Name=value syntax for passing Name-Value arguments to functions.
• General switch in many functions to fast thresholding.
• Switching away from using nanmean to mean('omitnan'), to be completed in next release.
• Improved loading of CIAtah directory (with ciapkg.loadBatchFxns).

New functions

• ciapkg.bf.getSeriesNoFromName - Returns series number for Bio-Formats series with name given by user.
• ciapkg.bf.dispFileSeries - Returns series number for Bio-Formats series with name given by user.
• writeMatfileToHDF5 - Writes a MAT-file dataset to HDF5 by reading the matrix in parts to avoid large overhead.
• mergeStructs - Copies fields in fromStruct into toStruct, if there is an overlap in field names, fromStruct overwrites toStruct unless specified otherwise.
• writeDataToMatfile - Converts 3D movie matrix stored in a file (HDF5, TIF, AVI, NWB) into a MAT-file for use with larger-than-memory processing.
• getNoRMCorreParams - Automatically returns default NoRMCorre parameters or ask user to input parameters.

Updated @ciatah

• ciatahMainGui - Improve speed of movie display by better passing of movie information to ciapkg.io.readFrame. If processed and raw movie are the same length, sliders move in sync. By default use 'painters' renderer as that can produce smoother rendering than opengl. Added file sizes to the folder files preview. Reset movie dimension on folder select change so movie previews run without outputting initial error (movies would still preview, users just might be confused by the warning). Feature to allow preview of HDF5 datasets. Expanding to Bio-Formats and other types of data with internal structures for data. Added ability to open folder in OS GUI, e.g. Windows Explorer. If manual input for filtering folders set back to no filter to avoid GUI loops. Make sure preview data and open folder buttons are associated with correct previewDataHandle and openFoldersHandle handles.
• getRegistrationSettings - Add additional black background support. Added largeMovieLoad setting.
• modelAddNewFolders - Add additional check for when user adds blank folder paths (NOT empty folders) and remove them. Empty folder check update.
• modelExtractSignalsFromMovie - CELLMax support for loading prior settings. Update to EXTRACT support: added additional options, do not automatically eliminate summary section, and more. Switch order of mergeStructs and supplying cell radius to EXTRACT, else empty vector can be passed depending on user input.
• modelGetSignalsImages - Ensure when loading cell extraction CIAtah-style that only MAT files are loaded.
• modelModifyRegionAnalysis - Misc code standard updates.
• modelPreprocessMovieFunction - Change all nested and local functions to start with "nestedfxn_" or "localfxn_" prefixes for clarity. Added option to fix "broken" frames, e.g. frames in which the camera gives out garbled or improperly formatted data. Fix with mean of the movie to reduce effect on downstream analysis. Disable local function "getMovieFileType" in favor or CIAtah package function. Added option to place prefix on all output files. Detrending now does everything within nested function instead of calling normalizeMovie() to reduce memory overhead. Added support for MAT-file based processing of movies. Add additional black background support. Added largeMovieLoad setting support, improve memory allocation for select movies. Make sure readFrame is compatible with all HDF5 dataset names. NormCorre integrated into the pipeline for end users.
• modelSaveImgToFile - Update PaperPosition to auto to avoid error "Positioning Figure for ResizeFcn. Set PaperPositionMode to 'auto' (match figure screen size) to avoid resizing and this warning." Further change to PaperPositionMode to avoid errors.
• modelVarsFromFiles - Enforce that NWB loading only looks for .nwb files and ensure that manual or automatic classifications are loaded when loading NWB files. Ensure when loading cell extraction CIAtah-style that only MAT files are loaded.
• viewMovieRegistrationTest - Updated so more settings are pulled from CIAtah object settings.
• viewSubjectMovieFrames - Update code standards.

Updated +ciapkg

• signalSorter - Misc updates.
• downloadCnmfGithubRepositories - Update URLs to include a backup URL for cvx since would often timeout.
• appendDataToHdf5 - comments updated.
• downsampleHdf5Movie - Added Bio-Formats support. Switch chunking to automatic by default, generally improved reading from data later than small chunks. Improved Name-Value comments.
• readHDF5Subset - minor updates.
• computeImageFeatures - Make default thresholding fast thresholding. Improved annotation of code and added options.fastThresholding. Also by default plots are not made. Refactored extra features code and finding centroid to avoid unnecessary function calls, speeding up code.
• filterImages - Make default thresholding fast thresholding.
• findCentroid - Make default thresholding fast thresholding. Update to ensure threshold based on max value if user inputs non-normalized (e.g. distribution forced to zero to one) thresholded images.
• groupImagesByColor - Added fast thresholding option and other threshold image options.
• thresholdImages - manageParallelWorkers now passed options.waitbarOn value to reduce command line clutter if user request in thresholdImages.
• getFileInfo - Add trial # to output.
• getFileList - Improved support multiple directory input. Added folder filter.
• getMovieFileType - Added MAT-file support. Add SlideBook Bio-Formats support.
• getOptions - In 2021a MATLAB introduced Name=value syntax for passing name-value arguments. These are passed within varargin as a string array instead of a char array as was the case with comma-separated syntax. getOptions now checks for isstring in addition to ischar to add support for this syntax to CIAtah functions. See https://www.mathworks.com/help/matlab/release-notes.html?rntext=&startrelease=R2021a&endrelease=R2021a&groupby=release&sortby=descending&searchHighlight=#mw_77c0932f-4a31-44e6-b550-db5a736c2de3.
• load_tif_movie - Speed improvements related to imfinfo calls. Added tiffread support and ScanImageTiffReader support (not user accessible currently).
• loadDependencies - Added PatchWarp (https://github.com/ryhattori/PatchWarp). Code clean up. Updated NoRMCorre to use https://github.com/bahanonu/NoRMCorre as that is a package version, cleaner namespace. Added CIAtah utilities repository (https://github.com/bahanonu/ciatah_utils) to separate outside code from main repository. Add SlideBook .jar reader to Bio-Formats download. Ensure NoRMCorre is in default options. Ensure bfmatlab_readers directory exists, else websave errors occur.
• loadMovieList - Speed improvements related to imfinfo calls. Add option to load movie into a MAT-file. Improve Bio-Formats support, including using bfGetPlane to partially load data along with adding ndpi support. Add SlideBook Bio-Formats support. Remove local function getMovieFileType to force use of CIAtah getMovieFileType function. Updated getIndex call to avoid issues with certain Bio-Formats. Ensure getting correct Bio-Formats frames and loadMovieList output. Hide TIF warnings.
• loadSignalExtraction - Added option to load manual sorting as well and output as standard vector.
• manageParallelWorkers - Added displayInfo option.
• readFrame - Ensure subfxn_loadFrame nested function outputs frame data (even if file does not contain the requested frame, e.g. empty matrix).
• saveMatrixToFile - Use Fast_Tiff_Write to write out TIF files instead of saveastiff by default. Give option with options.tifWriter.
• computeManualMotionCorrection - Check if inputs are sparse, convert to single. By default use 'painters' renderer as that can produce smoother rendering than opengl. Update to make sure titles update correctly for test images. Add renderer option to allow users to choose painters or opengl.
• turboregMovie - Refactor code to remove need to transform movie into cell array, performance and memory improvements. Added NoRMCorre support, function will eventually be merged with "registerMovie" or renamed to indicate support for multiple algorithms. Reason to integrate NoRMCorre into this function is takes advantage of existing pre-processing pipeline and integration with other algorithms (e.g. cross-session). Check if inputs are sparse, convert to single. Use custom bahanonu NoRMCorre that is within a package and update to use reference picture instead of template. Updated to which('normcorre.normcorre') from which('normcorre') since normcorre now inside a package. Additional NoRMCorre support and getNoRMCorreParams checking before running NoRMCorre.
• downsampleMovie - Update information for users/comments.
• normalizeMovie - Switch away from using nanmean to mean('omitnan'). Detrend flight code refactor.
• computeCnmfeSignalExtraction_batch - Updated to add option to ignore remove_false_positives.
• computePeakStatistics - Update comments along with code standard improvements. options.testpeaksArray can be used regardless of vector (with peak frames) orientation.
• computeSignalPeaks - Better comments for options.
• computeSignalSnr - Better comments for options.
• createMovieFromVector - Fixed issue where the min 2nd dimension would always be options.windowSize, leading to errors.
• changeFont - Also check for matlab.ui.control.UIControl when conducting font color changes and ignore to not cause errors.
• createGroupColorMaps - Added fast thresholding option.
• playMovie - AVI now read(...,'native') is faster. Also add primaryTrackingPointNback option to allow a trailing number of points from prior frames to appear. Added support for read from disk (minimal RAM use) matfile reading of a MAT-file with data in a specified variable name. Default background and theme is now black. By default use 'painters' renderer as that can produce smoother rendering than opengl. Sub-sampling now has option to downsample instead of just taking every X pixel, slower but better quality. Add renderer option to allow users to choose painters or opengl. Misc other changes and additional options.
• exampleFxn - Added nested and local functions to the example function.
• loadBatchFxns - Added patchwarp to list of excluded by default folders for external programs. Directly allow use of ciapkg.io.getOptions since it is not dependent on loadBatchFxns due to being inside a package. Also allow option for user to specify which special packages that are not normally loaded to be loaded rather than loading everything if they want a specific package loaded. Speed up removal of directories not needed by default by directly locating the root path of that algorithm rather than searching for the M-file within the entire external programs directory, saves time. Users must have downloaded dependencies using loadDependencies. Add SlideBook JAR to the path for Bio-Formats support.

---

This discussion was created from the release Code and GUI improvements, NoRMCorre and additional file formats support, etc..