NEWS

New in Version 2.0.9
====================
* Pydpiper now depends on `qbatch` (and hence a recent `gnuparallel`).
  One can configure queueing options through either qbatch or Pydpiper; see the INSTALL file.
  The 'headless' (`--submit-server`) mode hasn't been ported and still uses `qsub` directly.
* the above change adds SLURM scheduler support
* modifications to the supplied config files due to the above
* backport mincaverage copy_header fix from develop

New in Version 2.0.8
====================
* add option to perform atlas-to-atlas registration of the study model to a 'common space model' and resample images
  and determinants into this common space (MBM.py only).  The relevant flags are --(no-)common-space-registration,
  --common-space-model, --common-space-mask.  At the moment the registration parameters are automatically taken from the
  model-building LSQ12 and (final generation of) nonlinear registration settings
* add extra columns to output CSVs: common space images/transforms/determinants if created; input images in various 'spaces'

New in Version 2.0.7
====================
* this release fixes only one issue: minctracc-based model building didn't use previously computed transforms
  as initial transforms in the next level of registrations

New in Version 2.0.6
====================
* fix MAGeT extra resampling of labels along transform from original onto new templates (#304)
* MAGeT internals rewrite/simplification
* MBM and MAGeT can read files and masks from csv file containing `file` and optionally `mask_file` columns
* improved minctracc parameters and defaults for stiffness, weight, and lattice_diameter;
  max_def_magnitude has been removed (it no longer appears in minctracc's -help)
* additional determinism to allow restarting pipelines without losing stages
* --mem-request-attribute can now take a comma-separated list of attributes (e.g., "vmem,mem"),
  all of which currently get the same values
* --max-templates now includes the templates from the original library; this might be reverted soon
* the default blur level for the determinants is now 0.2mm only;
  specify additional/different levels via --stats-kernels, e.g., '0.1,0.2,0.5' (the old default)
* switched from mincANTS to ANTS (but symlinking to mincANTS is possible)
* the installation instructions now require minc-toolkit-v2 rather than minc-toolkit due to use of ANTS
* ANTS now uses the source ("fixed") instead of target ("moving") mask
* added some progress information in "verbose" mode
* improved (too few) filenames and locations
* some improved error messages
* numerous internal changes

New in Version 2.0.5
====================
* pairwise lsq12 now (correctly) includes a registration to "itself" for each subject.

New in Version 2.0.4
====================
* execution bugfixes

New in Version 2.0.3
====================
* better montage file locations/naming
* bugfix in "pride of models" association of models to timepoints
* execution bugfixes

New in Version 2.0.2
====================
* automatically set MAGeT's protocols based off model building LSQ12/nlin protocols when possible
* two-level pipeline supports multiple initial models
* when LSQ6 is disabled, input files are resampled to a common grid so various programs (e.g., mincaverage) will work
  (currently implemented in MBM, hence also in two-level, rather than in LSQ6 code itself)
* zip_safe flag disabled in `setup` so that config files will be installed in unzipped form
* more information is printed about stages unable to run due to memory limits
* fix hang running executors locally
* allow specification of objective function for linear minctracc stages
* slight fix to ANTS stage formatting allowing compatibility with minc-toolkit v2
* update SickKids HPF config file
* various internal code improvements

New in Version 2.0.1
====================
* made the lsq12 protocols required in order to provide proper error messages when not specified
* allowed the specification of a mincANTS protocol with a single level in lsq12_nlin which is used for MAGeT (among others)

New in Version 2.0
==================
* switch from Python2 to Python3.5 or higher
* PydPiper2.0 is a rewrite of PydPiper to facilitate easier creation of pipelines
* we've made significant changes in how the communication between server and executors work, and most network related issues appear to be resolved.
* new pipelines: asymmetry.py which is a pipeline that runs a twolevel_model_building on the input brains and their flipped versions
* MBM and twolevel_model_building have a build-in MAGeT option
* registration_chain code allows you to specify a "pride of models"; this is necessary for instance when there are large size differences between your timepoints
* you can provide your csv that you'll use for your entire project (and stats) and feed that into: MBM, registration_chain, twolevel_model_building
* usage differences:
  - must specify --files for MBM.py
  - some protocol defaults not specified; you need to specify these


New in Version 1.18.1
=====================
* point to ConfigArgParse 0.11.0 on github until its maintainer makes a new release and uploads to PyPI

New in Version 1.18
===================
* bug fix to minctracc memory config
* added a `--use-dir-names` flag to two-level code to create groups based on directory names
  rather than first input files
* update the dependencies; in particular, specify minimum ConfigArgParse version due to bug in that library
* many changes supporting operation on the Sick Kids CCM grid (and other PBS/Torque 'capability'
  systems) - mostly slowly implementing a HPC queue submission library inside Pydpiper ...
* new config file for CCM cluster
* added a --submit-server flag so that use of this mode of operations isn't determined by queue type
  (at the moment this only works for PBS/Torque grids)
* added some logic to executor `mainLoop` procedure to detect Pyro4 timeouts ... this doesn't ever
  seem to be triggered, so should probably be removed again in future
* fixed a bug where only --lsq12-max-pairs images were used to generate the lsq12 average;
  doesn't seem to have made much difference
* runnable-time hooks are now cached
  (and are now added via a method, rather than by appending to runnable_hooks directly)
* very hackish memory estimates for pmincaverage (for these to agree with SGE's estimates,
  probably need minc-stuffs 0.1.14 or later)
* fewer executors are launched in some situations when more aren't needed
* qsub return codes are checked
* job arrays are used to submit executors to avoid choking the grid engine and to reduce
  vmem usage of the server when submitting; query these with "qsub -t <jid>[]"
* Pyro knows how to serialize more exceptions
* removed the executor heartbeat thread; may be restored in future
* use subprocess32 instead of subprocess for (hopefully) improved correctness


New in Version 1.15
===================
* some of the python dependencies are now installed automatically (from PyPI): 
  ConfigArgParse, networkx, pygraphviz, Pyro4, pytest, pydot
* default value for the lsq12-max-pairs is set to 25
* the LSQ6.py module now has the ability to copy the header information of the 
  first input into the lsq6-average file
* the non-linear modules are hard coded to use pmincaverage instead of mincaverage 
  (for the time being)
* updates to the code to speed up the server process (less disk access when creating 
  directories, and directories are crated only once)
* new SciNet default (in the configuration file): latency tolerance is set to 10 
  minutes (delay between executor-server pings), executor-start-delay is set to 
  10 minutes: the local executor starts 10 minutes after the server to make sure 
  that the server is ready/created the uri, otherwise the executor dies right away, 
  and no processing will be done on the compute node that runs the server.
* the server logs its own memory usage, and uses this information when launching 
  local executors
* changed the graph representation to save memory/time when computing newly runnable 
  stages after a stage completes
* opened up an argument for the executors to specify which pe (parallel environment) 
  is used on SGE
* executors shutdown when they are unable to communicate with the server. This is to 
  avoid having them sit idle when the server is gone
* removed the deprecated options --queue and --sge-queue-opts
* added the option --mem-request-variable which allows you to change the memory request 
  variable to use for sge/ogs


New in Version 1.14
===================
* the build model and registration chain applications now both create a
  composite image of the input files and print a message to the user,
  allowing them to check that the orientation is sufficiently similar
  that the LSQ6 registration method chosen is likely to succeed
* memory requirements for mincblur and nonlinear minctracc stages are now
  estimated dynamically before such a stage runs.  The nlin minctracc protocol
  now accepts a "memory" parameter allowing you to override the per-voxel
  default (based on the native-resolution call) with a fixed amount in GB
  per minctracc generation, e.g.: '"memory";0.5;1;1;1.5;2' dependending
  on the particularities of your images or to get better estimates than those
  based on the finest resolution
* due to poor performance on SciNet, blur calls running on PBS queues now
  use pmincaverage instead of mincaverage.  This adds a dependency on pyminc;
  see the INSTALL file.
* PydPiper's understanding of the "gradient" variable in the minctracc
  nonlinear protocols and the internal `useGradient` variable have changed:
  "FALSE"/False (resp.) causes the blurring to be applied to the image intensities;
  "TRUE"/True causes it to be applied to the image gradients, vs. the previous
  behaviour (True implied blurring of *both* the intensities and the gradient)
* changes to the supplied minctracc nonlinear protocol, partly to reflect the
  above gradient-related changes
* the registration chain now allows you to specify --average-time-point=-1
  to use the last set of scans for inter-subject registration
* additional checking of input files
* a test script which exercises the MBM and MAGeT applications is
  now provided with instructions in pydpiper_testing/
* SGE executors now request memory dynamically based on the requirements of
  the runnable stages.  In this configuration, the --mem option now gives
  a maximum amount allowed per executor.  The system exits if jobs require
  more memory than this maximum.
* on PBS, the remote server now writes its stdout to a text file in the working
  directory, so the user needn't dig into, e.g., SciNet's /var to examine
  it while the pipeline is still running
* a few changes to defaults: default memory is now 1.75G/job rather than 2;
  PBS executors wait for up to 3 minutes for the server to start instead of 50s;
  a few changes to the supplied config files
* better handling of csv files with extra commas in the registration chain
* various bugfixes, in particular to the registration chain
* some improved logic w.r.t. updating state of stages after failure of a
  dependency
* the internal queue has been changed to a set to support additional operations
  related to scheduling and resource management (particularly the dynamic
  executor memory requests; see above)
* miscellaneous code improvements
* *** *** BETA 2 *** ***
* fixed bug in determining the max filename length for mincblur
* added the "alternate input" functionality (using a separate data set for
  the LSQ6 stage, then apply the transformations to the original data set
  and continue with that)
* better handling of options for qsub
* added ability to set stage memory default
* added the --greedy flag to force executors to use the alloted memory
* fixed pmincaverage call


New in Version 1.13
===================
* a composite image is now created after the LSQ6 stage and a message output,
  allowing the user to abort the pipeline if the registration has failed
* reverted back to launching executors without syncwith (which didn't work)
* memory usage for mincANTS commands is determined by the code based on total
  number of voxels in the files (can be overridden using the non linear protocol)
* the command stage type now has runnable-time and completion-time hooks, allowing
  arbitrary Python code to be run (this is used in the implementation of the memory
  usage and composite image features above)
* a verification image is created after the LSQ6 stage
* the registration chain works (but has not been tested on its full functionality)
* fixed a bug that removed part of the filename for *final-nlin* files if the 
  original files contained a dot 

  (NOTE: this breaks compatibility with version 
  1.12, because the output filenames are now changed)

* heartbeat counts are stored in order to debug latency issues between the server
  and executors

New in Version 1.12
===================
* LSQ6 (MBM) fixes to the combination of --no-nuc and --inormalize. This would incorrectly apply the lsq6 transformation to the inormalized file twice
* MAGeT -- now uses the pipeline_name argument -- has better error messages when there are issues with the atlases/labels/masks -- if no atlases are found in the atlas library, the program exists -- has an explicit way to specify --pairwise
* improvements to error handling in the code
* most pipeline related files and directories are now prefixed with the pipeline name
* fixes to how the initial model is used. Now there are two possible scenarios:

  image.mnc
  image_mask.mnc

  or

  image.mnc
  image_mask.mnc
  image_native.mnc
  image_native_mask.mnc
  image_native_to_standard.xfm

* debug statements added in order to inspect heartbeat issues -- heartbeat checking can also be turned off at the moment
* on SciNet, as many executors as possible will start with the server using syncwith rather than after (in qsub)

New in Version 1.11
===================
(in addition to the changes in the alpha version)
* MAGeT fix: the argument max-templates was incorrectly used, and meant in 
  practice that atlases/segmentations were only created for the first 25 input 
  files (default for max-templates). This is now fixed.
* added the bootstrap option to MBM: use the first input file to the pipeline as 
  the target for the LSQ6 stage
* fixed the lsq6-target option for MBM: specify a single external file that will 
  be used as the target for the LSQ6 stage
* The parameters for some stages (mincblur/minctracc) are based on the resolution 
  of the input files. When the stage graph is created prior to running the pipeline, 
  this resolution is not always known and we used the resolution of the original input 
  files. At run time, the resolution sometimes changed, creating discrepancies between 
  stages at graph creation and run time. This would cause these stages to be rerun. 
  This is now fixed. The resolution of the files are based predominantly on the initial 
  model files or the target that is used for the LSQ6 stage.
* fixed a masking issue in MAGeT: it would produce output files with an image range of 
  unsigned short 0 to 1. The masked files now have a proper image range.
* mincblur has a hardcoded limit in terms of its output file. At graph creation time the 
  pipeline now checks to see whether mincblur can actually be run. If not, the pipeline 
  won't start and advice you to either shorten the input file names or change the path 
  you are running the pipeline in.
* Added standard configurations for pipelines (MICe, SciNet), these are accessed through 
  the environment variable PYDPIPER_CONFIG_FILE
* file containing the command that was run and the version of pydpiper being used is written out
* instead of pickling the pipeline status (which wasn't happening anymore), hashes of 
  finished jobs are written out to a finished stages file
* the default mincANTS parameters have changed from:
  [SyN 0.5, Gauss 5 1, 100x100x100x0], [SyN 0.4, Gauss 5 1, 100x100x100x20], [SyN 0.4, Gauss 5 1, 100x100x100x50]
  to:
  [SyN 0.1, Gauss 2 1, 100x100x100x0], [SyN 0.1, Gauss 2 1, 100x100x100x20], [SyN 0.1, Gauss 2 1, 100x100x100x100]
* lsq12-max-pairs is now implemented

New in Version 1.11-alpha
=========================
* Moved from optparse to ConfigArgParse, which must be installed.
  See README for usage details.
* Improve server logic for handling failed executors and cases where
  executors have insufficient resources to run available jobs.
* Improved logic for handling walltime restrictions on PBS queues.
  New flags are available to set some PBS-related variables.
* Load correct modules on SciNet, with better error reporting
  from module load failures and xfmavg stages.
* To replace the old pickling of the current system state, there's
  a new mechanism for logging completed stages to disk
  (in pydpiper-backups/finished-stages) and correctly restarting the pipeline.
* Fixed a bug where executor threads spawned from the server could hang.
* Various small bugfixes, internal improvements, changes to logging and
  usage messages, and updates to comments.
* Added a .gitignore.

New in Version 1.10
===================
* PYRO environment variables are handled inside the code. The SERVERTYPE is 
  set statically to multiplex (there are potential issues with the thread option)
  and the LOGLEVEL and DETAILED_TRACEBACK can be overridden by environment 
  variables (see the README)

New in Version 1.9
==================
* MAGeT uses the originally optimized defaults again. Hardcoded paths to the 
  linear and non linear protocol for now. With a help message of where to find
  them when not running the code at MICe.

New in Version 1.8
==================
* major code restructuring on the server and executors. The executors now send
  out a heartbeat signal to the server, which enables the server to notice 
  an executor having died. Also, all executors now properly die when the server
  is done (or gets killed).
* pickling of the pipeline has been disabled, because it turned out to cause
  huge communication delays (minutes in large pipelines: > 30000 stages)
* command line tool added to query the server as to what's happening:
  check_pipeline_status.py
* logging is separated out again (each executor has its own log file)
* Pyro4 environment variables should be set as:
   
  PYRO_DETAILED_TRACEBACK=True
  PYRO_SERVERTYPE=multiplex
  PYRO_LOGLEVEL=INFO


New in Version 1.7
==================
* the communication between the server and the executors has been upgraded
  from Pyro3 to Pyro4. Currently the following environment variables should
  be set:
  
  PYRO_DETAILED_TRACEBACK=True
  PYRO_COMMTIMEOUT=2
  PYRO_SERVERTYPE=multiplex
  PYRO_LOGLEVEL=DEBUG
  
* the stiffness, weight and similarity parameters for minctracc can now be
  set through the non linear protocol in MBM.py 
* all output from the server/executors will be stored in the current 
  working directory


New in Version 1.6
====================
* fix to MAGeT.py. Solves issues with the voxel_vote command. 

New in Version 1.5
====================
* added retrying/resubmitting of failed files to the server. At this moment
  each stage is resubmitted twice before considering it truly failed. This solves
  an apparent nfs race condition issue.

New in Version 1.4
====================
* added a warning statement when the server is finished running and shutting
  down. There is still a communication issue, and sometimes executors do not
  shut down correctly.
* added verbosity option to the server. By default is now less verbose
* changed the output of the server to the user. Now displays more meaningful
  information: total number of stages to be run, and the number of stages that
  have already been processed.
* bug fix in the pipeline executor: can now be launched from the command line
  again

New in Version 1.3
====================
* huge amounts of code restructuring
* server now launches and relaunches executors (keeps track of executors that
  are registered with the server. However, for now does not check whether the 
  registered executors are alive once in a while)
* executors can kill themselves when they've been idle for too long. They can also
  "drain" themselves (not accept any new jobs) after a certain time and then 
  shutdown when the running jobs have finished.
* added an interrupt handler for ctr+c to the server and executors
* executors on shutdown kill the running processes by using their process IDs (pid)
* a whole bunch of changes to the twolevel_model_building code... (see GitHub)
* a whole bunch of changes to the registration_chain code... (see GitHub)