HOMER
                  Helper Of My Eternal Retrievals
===============================================================================


Author :       Michael D. Himes    University of Central Florida
Contact:       [email protected]

Advisor:       Joseph Harrington   University of Central Florida

Contributors:  David C. Wright
                 - containerized the code
               Zaccheus Scheffer
                 - assisted with refactoring the codebase that led to HOMER


Acknowledgements
----------------
This research was supported by the NASA Fellowship Activity under NASA Grant 
80NSSC20K0682.  We gratefully thank Nvidia Corporation for the Titan Xp GPU 
that was used throughout development of the software.


Summary
=======
HOMER is a Bayesian inverse modeling code.  Given some data and 
uncertainties, the posterior distribution is determined for some model. 

HOMER uses the Large-selection Interface for Sampling Algorithms (LISA) for 
its Bayesian framework.  LISA allows for both MCMC and nested sampling 
algorithms.  For details, see the LISA User Manual at 
https://exosports.github.io/LISA/doc/LISA_User_Manual.html.

HOMER's forward model is a neural network (NN) surrogate model trained by 
MARGE.  For details on MARGE, see the MARGE User Manual at 
https://exosports.github.io/MARGE/doc/MARGE_User_Manual.html.

HOMER comes with complete documentation as well as a user manual to assist 
in its usage.  Users can find the latest HOMER User Manual at 
https://exosports.github.io/HOMER/doc/HOMER_User_Manual.html.

HOMER is an open-source project that welcomes improvements from the community 
to be submitted via pull requests on Github.  To be accepted, such improvements 
must be generally consistent with the existing coding style, and all changes 
must be updated in associated documentation.

HOMER is released under the Reproducible Research Software License.  Users are free to 
use the software for personal reasons.  Users publishing results from HOMER 
(or modified versions of it) in a peer-reviewed journal are required to publicly
release all necessary code/data to reproduce the published work.  Modified 
versions of HOMER are required to also be released open source under the same 
Reproducible Research License.  For more details, see the text of the license.


Files & Directories
===================
HOMER contains various files and directories, described here.

doc/            - Contains documentation for HOMER.  The User Manual contains 
                  the information in the README with more detail, as well as a 
                  walkthrough for setup and running an example.
environment.yml - Contains all required packages (w/ versions) for HOMER.
example/        - Contains example of executing HOMER.
                  Includes example functions for each sampling algorithm.
                  See README within the directory for more details.
HOMER.py        - The executable driver for HOMER. Accepts a configuration file.
lib/            - Contains the classes and functions of HOMER.
  bestfit.py    - Contains functions to plot the best-fit spectrum.
  compost.py    - Contains functions to compare posterior distributions.
  func.py       - Contains example functions to be evaluated by an MCMC sampler.
  NN.py         - Contains a function to load a MARGE NN model.
  plotter.py    - Contains functions to plot results of the inference.
  utils.py      - Contains utiity functions.
Makefile        - Handles setting up BART, and creating a TLI file.
modules/        - Contains submodules for data generation, quantile calculation.
  datasketches  - Streaming quantiles package.
  LISA          - Bayesian sampler package.
README          - This file!


Note that all .py files have complete documentation; consult a specific file 
for more details.


Installation
============
After recursively cloning the repo, users must compile one of LISA's modules. 
From HOMER's directory,
    make lisa

If the user wishes to compute median and 1-2-3sigma spectra, they will require 
the datasketches package.  From the HOMER directory, enter
    make datasketches
into the terminal.  Note that this package is NOT required to run HOMER.

To install both packages easily, do
    make all


Executing HOMER
===============
HOMER is controlled via a configuration file.  After setting up the 
HOMER configuration file, run HOMER via
    ./HOMER.py <path/to/configuration file>

Note that HOMER requires a trained MARGE model.  For details, see the MARGE 
User Manual (http://planets.ucf.edu/bart-docs/MARGE_user_manual.pdf).


Setting Up a HOMER Configuration File
=====================================
A HOMER configuration file contains many options, which are detailed in this 
section.  For an example, see config.cfg in the example subdirectory, and the 
associated README with instructions on execution.


Directories
-----------
inputdir  : str.  Directory containing HOMER inputs.
outputdir : str.  Directory containing HOMER outputs.


Run Parameters
--------------
quantiles   : bool. Determines whether to compute quantiles.
                    If the Datasketches library is not installed, this setting 
                    has no effect.
onlyplot    : bool. Determines whether to skip the inference.
                    Reproduces plots \& calculations related to posterior.
plot_PT     : bool. Determines whether to compute the explored 
                    pressure--temperature profiles using the formulation of 
                    Line et al. (2013).  Presently, this requires the model 
                    parameters to be ordered as 
                    log(kappa), log(gamma1), log(gamma2), alpha, beta, 
                    followed by any other parameters.
credregion  : bool. Determines whether to calculate the 68, 95, and 99\% 
                    credible regions \& uncertainties.
compost     : bool. Determines whether to compare HOMER's posterior to 
                    another.
compfile    : str.  Path to posterior .npy file to compare with HOMER.
compname    : str.  Name of the other posterior for plot legends.
compsave    : str.  File name prefix for the saved comparison plots.
compshift   : floats. Shifts all values of a particular parameter by a 
                    set amount in the posterior to be compared, such as 
                    for factor-of-10 unit conversions in a log space.
                    Format: val1 val2 val3 val4 ...
                    E.g., compshift = 1 0 0 would increase all of the first 
                    parameter's values by 1 and leave the other parameters 
                    alone.
postshift   : floats. Same as `compshift`, but for HOMER's posterior.


Data Normalization Parameters
-----------------------------
ilog        : bool. Determines whether the NN takes the logarithm of the 
                    inputs.
                    Alternatively, specify comma-, space-, or newline-separated 
                    integers to selectively take the log of certain inputs.
olog        : bool. Determines whether the NN predicts the log of the 
                    outputs.
                    Alternatively, specify comma-, space-, or newline-separated 
                    integers to selectively take the log of certain outputs.
normalize   : bool. Determines whether to standardize the data by its 
                    mean and standard deviation.
scale       : bool. Determines whether to scale the data to be within a 
                    range.
scalelims   : ints. Range to scale the data to.
                    Format: low, high
fmean       : str.  Path to .NPY file of mean training input and output 
                    values.
                    Format: [inp0, inp1, ..., outp0, outp1, ...]
                    If relative path, assumed to be with respect to the 
                    input directory.
fstdev      : str.  Path to .NPY file of standard deviation of 
                    inputs/outputs.
                    See `fmean` for format \& path description.
fmin        : str.  Path to .NPY file of minima of inputs/outputs.
                    See `fmean` for format \& path description.
fmax        : str.  Path to .NPY file of maxima of inputs/outputs.
                    See `fmean` for format \& path description.


Neural Network (NN) Parameters
------------------------------
weight_file: str. File containing NN model & weights.
                  NOTE: MUST end in .h5
inD  : int. Dimensionality of the input  to the NN.
outD : int. Dimensionality of the output of the NN.


Bayesian Sampler Parameters
---------------------------
alg         : str.  Bayesian sampling algorithm.  Options:
                      demc                 ter Braak (2006)
                      snooker      ter Braak & Vrugt (2008)
                      multinest         Feroz et al. (2008)
                      ultranest              Buchner (2014, 2016, 2019)
func        : strs. Function and file to evaluate at each iteration of 
                    the MCMC.
                    Format: 
                      function file
                    or, if the file is in a different directory,
                      function file path/to/file/
                    Note: omit the '.py' from `file`.
pnames      : strs. Name of each free parameter. Can include LaTeX 
                    formatting.
pinit       : floats. Initial parameters for the MCMC.
pmin        : floats. Minima for free parameters.
pmax        : floats. Maxima for free parameters.
pstep       : floats. Step size for free parameters. 
                      This will change throughout the MCMC due to the 
                      differential evolution algorithm used.
data        : floats. Values to be fit via MCMC. 
                    Format: Separate each value by an indented newline.
                    Alternatively, specify a .NPY file.
uncert      : floats. Uncertainties on values to be fit via MCMC. 
                      Same format (indented newlines or .NPY)


Additional Sampler Parameters
-----------------------------
Only required if your setup requires the parameter.

filters     : strs. Paths to filter bandpasses associated with each datum. 
                    Separate each by indented newlines.
                    X-axis values must have the same type of units as `xvals`,
                    but may be separated by a constant multiplicative factor
                    (e.g., mm vs um, ft vs inch), see `filtconv` argument.
starspec    : str.  Path to .NPY file of the stellar spectrum.
                    More generally, this is an array of scaling factors that 
                    can be used when calculating the model.
factor      : str.  Path to .NPY file of scaling factor by which 
                    to modify de-normalized predictions. 
                    E.g., unit conversion.
wn          : bool. In astro context, determines whether the X-axis units are 
                    spatial frequency (True) or wavelength (False).
                    More generally, when wn = False, the X-axis values are 
                    transformed to their inverse when plotting and 
                    band-integrating models.  
wnfact      : float. Multiplication factor when computing the inverse of 
                     `xvals`.
                     E.g., if `xvals` is in cm-1 and the desired inverse units 
                     are um, then wnfact = 1e4.
filtconv    : float. Multiplication factor to convert the filter 
                     X-value units to the `xvals` units.
                     E.g., if filter X values are in nm, but `xvals` are in um, 
                     this argument would be set to 1e-3.


MCMC Parameters
---------------
Not required for nested sampling algorithms.

flog        : str.  Path to MCMC log file. 
                    If relative, with respect to input dir.
nchains     : int.  Number of parallel samplers.
niter       : int.  Number of total iterations.
burnin      : int.  Number of burned iterations from the beginning of 
                    chains.
thinning    : int.  Thinning factor for posterior.


Plotting Parameters
-------------------
xvals      : str.  Path to .NPY file containing the x-axis values 
                   associated with a prediction.
                   If relative, path is with respect to `inputdir`.
xlabel     : str.  X-axis label for plots.
ylabel     : str.  Y-axis label for plots.
fpress     : str.  Path to text file containing the pressures of each 
                   layer of the atmosphere, for plotting T(p) profiles.
                   If relative, with respect to `inputdir`.
PTargs     : str.  Path to .txt file containing values necessary to 
                   calculate the temperature--pressure profile.
                   Currently, only option is Line et al. (2013) method.
                   Format: R_star (m), T_star (K), T_int (K), 
                           sma (m), grav (cm s-2)
                   If plot_PT is False, this argument does nothing.
savefile   : str.  (optional) Prefix for MCMC plots to be saved.


Versions
========
HOMER was developed on a Unix/Linux machine using the following 
versions of packages:
 - Python 3.7.2
 - Keras 2.2.4
 - Numpy 1.16.2
 - Matplotlib 3.0.2
 - mpi4py 3.0.3
 - Scipy 1.2.1
 - sklearn 0.20.2
 - Tensorflow 1.13.1
 - CUDA 9.1.85
 - cuDNN 7.5.00
 - ONNX 1.6.0
 - keras2onnx 1.6.1
 - onnx2keras 0.0.18
 - pymultinest 2.10
 - ultranest 2.2.2

See the supplied environment.yml file.


Be kind
=======
Please cite this paper if you found this package useful for your research:

Himes et al. 2022, PSJ, 3, 91
https://iopscience.iop.org/article/10.3847/PSJ/abe3fd/meta

@ARTICLE{2022PSJ.....3...91H,
       author = {{Himes}, Michael D. and {Harrington}, Joseph and {Cobb}, Adam D. and {G{\"u}ne{\c{s}} Baydin}, At{\i}l{\i}m and {Soboczenski}, Frank and {O'Beirne}, Molly D. and {Zorzan}, Simone and {Wright}, David C. and {Scheffer}, Zacchaeus and {Domagal-Goldman}, Shawn D. and {Arney}, Giada N.},
        title = "{Accurate Machine-learning Atmospheric Retrieval via a Neural-network Surrogate Model for Radiative Transfer}",
      journal = {\psj},
     keywords = {Exoplanet atmospheres, Bayesian statistics, Posterior distribution, Convolutional neural networks, Neural networks, 487, 1900, 1926, 1938, 1933},
         year = 2022,
        month = apr,
       volume = {3},
       number = {4},
          eid = {91},
        pages = {91},
          doi = {10.3847/PSJ/abe3fd},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2022PSJ.....3...91H},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Thanks!