Skip to content

Commit

Permalink
Merge pull request #27 from simonsobs/development
Browse files Browse the repository at this point in the history
Development
  • Loading branch information
mattyowl authored Jun 26, 2020
2 parents 6dd6b33 + bf71516 commit 1fb2192
Show file tree
Hide file tree
Showing 29 changed files with 1,700 additions and 239 deletions.
21 changes: 13 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
Nemo is a map filtering and source detection and characterization pipeline, designed to find
galaxy clusters using the Sunyaev-Zel'dovich effect. It can also be used to find sources.

* Documentation: <https://acru.ukzn.ac.za/~mjh/nemo/>
* Documentation: <https://astro.ukzn.ac.za/~mjh/nemo/>
* License: Will be BSD 2-clause when released (this is a currently a private repository: for
now Nemo cannot be used for non-ACT, non-SO approved projects; however, the aim is to make the
code fully public by the time the first AdvACT cluster catalog is published)
code fully public by the time the first AdvACT cluster catalog is published).
* Authors: Matt Hilton, with contributions from Simone Aiola, David Alonso, Matthew Hasselfield,
Toby Marriage, Sigurd Naess, and Cristóbal Sifón.

Nemo is *not* the pipeline used for [Hasselfield et al. (2013)](http://adsabs.harvard.edu/abs/2013JCAP...07..008H),
but implements many of the ideas presented there, and should give similar results, given
the same map (or at least it did in the past). *It is* the pipeline, that has been used for the
[two-season ACTPol cluster catalog paper](http://adsabs.harvard.edu/abs/2017arXiv170905600H).
[two-season ACTPol cluster catalog paper](http://adsabs.harvard.edu/abs/2017arXiv170905600H),
and the AdvACT S18 cluster catalog paper.

A slightly modified version of Matthew Hasselfield's `gnfw.py` code (used for modeling cluster
pressure profiles) is included in Nemo.
Expand All @@ -30,24 +33,25 @@ removed, except for the version used for the
# Software needed

Nemo itself is written in Python (3.6+), and requires the following additional modules to be installed
(currently used versions are given in brackets, earlier and later versions also probably work):
(currently used versions are given in brackets, later versions also probably work):

* numpy (1.13.3)
* scipy (1.3.0)
* matplotlib (2.1.1)
* astLib (0.11.3)
* [pixell](https://github.com/simonsobs/pixell/) (0.5.3 or git version)
* [pixell](https://github.com/simonsobs/pixell/) (0.6.3 or git version)
* Pillow (5.1.0)
* astropy (3.2.1)
* IPython (7.2.0)
* Cython (0.24.1)
* PyYAML (3.12)
* Colossus (1.2.9)
* [CCL](https://github.com/LSSTDESC/CCL) (2.1 or later)
* mpi4py (3.0.0)
* colorcet (1.0.0; https://github.com/bokeh/colorcet/releases)

All of the dependencies can be installed using `pip`, and should be installed automatically as needed
by the `setup.py` script.
by the `setup.py` script (your mileage may vary in terms of how successful `pip` is at building
some of the external dependencies, depending on your set up).

# Installation

Expand Down Expand Up @@ -82,7 +86,8 @@ export PYTHONPATH=$HOME/local/lib/python3.6/site-packages:$PYTHONPATH

# Running Nemo

Documentation is available at <https://acru.ukzn.ac.za/~mjh/nemo/>.
Documentation is available at <https://astro.ukzn.ac.za/~mjh/nemo/>, including a number of
tutorials.

See [examples/equD56/README.md](examples/equD56/README.md) for a tutorial on how to re-create
the ACTPol two-season cluster catalog (including mass estimates).
Expand Down
5 changes: 3 additions & 2 deletions bin/nemo
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,9 @@ if __name__ == '__main__':
H0=70.
Om0=0.30
Ob0=0.05
sigma_8=0.8
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma_8, enableDrawSample = True)
sigma8=0.8
ns=0.95
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma8, ns, enableDrawSample = True)
selFnCollection=pipelines.makeSelFnCollection(config, mockSurvey)
if config.MPIEnabled == True:
config.comm.barrier()
Expand Down
2 changes: 1 addition & 1 deletion bin/nemoCatalogCheck
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ if __name__ == '__main__':
configFileName=args.configFileName
catFileName=args.catalogFileName
config=startUp.NemoConfig(configFileName, setUpMaps = False, MPIEnabled = False, verbose = False)
outputLabel=os.path.split(parDictFileName)[-1].replace(".yml", "")
outputLabel=os.path.split(configFileName)[-1].replace(".yml", "")

print(">>> Checking catalog %s against nemo output:" % (catFileName))
optimalCatalogFileName=config.rootOutDir+os.path.sep+"%s_optimalCatalog.fits" % (os.path.split(config.rootOutDir)[-1])
Expand Down
24 changes: 5 additions & 19 deletions bin/nemoCosmo
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ import warnings
#warnings.filterwarnings("error")

#-------------------------------------------------------------------------------------------------------------
def lnprob(H0, Om0, sigma8, Ob0, tenToA0, B0, Mpivot, sigma_int):
def lnprob(H0, Om0, sigma8, Ob0, ns, tenToA0, B0, Mpivot, sigma_int):
"""Log likelihood function for use with Cobaya.
"""

# This can fail if wander outside parameter space where mass function is defined
try:
selFn.update(H0, Om0, Ob0, sigma8, scalingRelationDict = {'tenToA0': tenToA0, 'B0': B0,
'Mpivot': Mpivot, 'sigma_int': sigma_int})
selFn.update(H0, Om0, Ob0, sigma8, ns, scalingRelationDict = {'tenToA0': tenToA0, 'B0': B0,
'Mpivot': Mpivot, 'sigma_int': sigma_int})
except:
return -np.inf

Expand Down Expand Up @@ -152,8 +152,8 @@ if __name__ == '__main__':
['sigma8', {'prior': {'min': 0.6, 'max': 0.9},
'ref': {'dist': 'norm', 'loc': 0.8, 'scale': 0.1},
'proposal': 0.02,
'latex': '\sigma_8'}]
])
'latex': '\sigma_8'}],
['ns', 0.95]])
if args.dumpConfig == True:
cobaya.yaml.yaml_dump_file("default.yml", info, error_if_exists = False)
print("Dumped default Cobaya configuration to `default.yml`")
Expand Down Expand Up @@ -183,17 +183,3 @@ if __name__ == '__main__':
# Run Cobaya...
updated_info, products=run(info)

# We'll generally be running under MPI... so just run GetDist later instead...
#gd_sample=MCSamplesFromCobaya(updated_info, products["sample"])

#g=gdplt.getSubplotPlotter()
#params=['H0', 'Om0', 'sigma8']
#param_3d = None
#g.triangle_plot(gd_sample, params, plot_3d_with_param=param_3d, filled=True, shaded=False)
#g.export(fname=cosmoOutDir+os.path.sep+"cornerplot.pdf")
#g.export(fname=cosmoOutDir+os.path.sep+"cornerplot.png")

#mean=gd_sample.getMeans()[:3] # H0, Om0, sigma8
#print("Mean:")
#print(mean)

15 changes: 10 additions & 5 deletions bin/nemoMass
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,8 @@ if __name__ == '__main__':
if key not in tab.keys():
forcedPhotometry=True
config=startUp.NemoConfig(parDictFileName, MPIEnabled = args.MPIEnabled, divideTilesByProcesses = False,
setUpMaps = forcedPhotometry, verbose = False, strictMPIExceptions = strictMPIExceptions)
setUpMaps = forcedPhotometry, writeTileDir = False, verbose = False,
strictMPIExceptions = strictMPIExceptions)

# Remaining set-up
massOptions=config.parDict['massOptions']
Expand Down Expand Up @@ -314,11 +315,15 @@ if __name__ == '__main__':
Ob0=massOptions['Ob0']
else:
Ob0=0.05
if 'sigma_8' in list(massOptions.keys()):
sigma_8=massOptions['sigma_8']
if 'sigma8' in list(massOptions.keys()):
sigma8=massOptions['sigma8']
else:
sigma_8=0.8
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma_8)
sigma8=0.8
if 'ns' in list(massOptions.keys()):
ns=massOptions['ns']
else:
ns=0.95
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma8, ns)

colsToAdd=['M500', 'M500Uncorr', 'M200m', 'M200mUncorr',
'M500NoRel', 'M500UncorrNoRel', 'M200mNoRel', 'M200mUncorrNoRel']
Expand Down
100 changes: 100 additions & 0 deletions bin/nemoModel
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/usr/bin/env python

"""
Make a map containing model objects using the info in a nemo output catalog
"""

import os
import sys
import numpy as np
import astropy.table as atpy
from astLib import *
from nemo import startUp
from nemo import maps
import argparse
import astropy.io.fits as pyfits

#------------------------------------------------------------------------------------------------------------
if __name__ == '__main__':

parser=argparse.ArgumentParser("nemoModel")
parser.add_argument("catalogFileName", help = """A catalog (FITS table format) as produced by nemo.""")
parser.add_argument("templateFileName", help = """A FITS image file. The output sky model image will
have the same pixelization.""")
parser.add_argument("beamFileName", help = """A file containing the beam profile, in the standard format
used by ACT.""")
parser.add_argument("outputFileName", help = """The name of the output file that will contain the sky
model image.""")
parser.add_argument("-f", "--frequency-GHz", dest = "obsFreqGHz", type = float, default = 150.0,
help = """If the nemo catalog contains SZ-selected clusters, the SZ signal will be
evaluted at the given frequency, ignoring relativistic effects
(default: 150.0).""")
parser.add_argument("-M", "--mpi", dest="MPIEnabled", action="store_true", help="""Enable MPI. If used,
the image will be broken into a number of tiles, with one tile per process. If you
want to use this, run with e.g., mpiexec -np 4 nemoMass configFile.yml -M""",
default = False)
parser.add_argument("-n", "--no-strict-errors", dest="noStrictMPIExceptions", action="store_true",
help="""Disable strict exception handling (applies under MPI only, i.e., must be
used with the -M switch). If you use this option, you will get the full traceback
when a Python Exception is triggered, but the code may not terminate. This is due
to the Exception handling in mpi4py.""",
default = False)
args = parser.parse_args()

if args.noStrictMPIExceptions == True:
strictMPIExceptions=False
else:
strictMPIExceptions=True

# Create a stub config (then we can use existing start-up stuff to dish out the tiles)
parDict={}
mapDict={}
mapDict['mapFileName']=args.templateFileName
mapDict['obsFreqGHz']=args.obsFreqGHz
mapDict['beamFileName']=args.beamFileName
parDict['unfilteredMaps']=[mapDict]
parDict['mapFilters']=[]
if args.MPIEnabled == True:
parDict['makeTileDir']=True
parDict['tileOverlapDeg']=1.0
parDict['tileDefLabel']='auto'
parDict['tileDefinitions']={'targetTileWidthDeg': 10.0, 'targetTileHeightDeg': 5.0}

config=startUp.NemoConfig(parDict, MPIEnabled = args.MPIEnabled, divideTilesByProcesses = True,
makeOutputDirs = False, setUpMaps = True, writeTileDir = False,
verbose = False, strictMPIExceptions = strictMPIExceptions)

tab=atpy.Table().read(args.catalogFileName)

# Build a dictionary containing the model images which we'll stich together at the end
modelsDict={}
for tileName in config.tileNames:
shape=(config.tileCoordsDict[tileName]['clippedSection'][3],
config.tileCoordsDict[tileName]['clippedSection'][1])
wcs=astWCS.WCS(config.tileCoordsDict[tileName]['header'], mode = 'pyfits')
try:
modelsDict[tileName]=maps.makeModelImage(shape, wcs, tab, args.beamFileName,
obsFreqGHz = args.obsFreqGHz)
except:
raise Exception("makeModelImage failed on tile '%s'" % (tileName))

# Gathering
if config.MPIEnabled == True:
config.comm.barrier()
gathered_modelsDicts=config.comm.gather(modelsDict, root = 0)
if config.rank == 0:
print("... gathered sky model tiles ...")
for tileModelDict in gathered_modelsDicts:
for tileName in tileModelDict.keys():
modelsDict[tileName]=tileModelDict[tileName]

# Stitching
if config.rank == 0:
d=np.zeros([config.origWCS.header['NAXIS2'], config.origWCS.header['NAXIS1']])
wcs=config.origWCS
for tileName in modelsDict.keys():
minX, maxX, minY, maxY=config.tileCoordsDict[tileName]['clippedSection']
d[minY:maxY, minX:maxX]=modelsDict[tileName]
astImages.saveFITS(args.outputFileName, d, wcs)
5 changes: 3 additions & 2 deletions bin/nemoSelFn
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,9 @@ if __name__ == '__main__':
H0=70.
Om0=0.30
Ob0=0.05
sigma_8=0.8
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma_8, enableDrawSample = True)
sigma8=0.8
ns=0.95
mockSurvey=MockSurvey.MockSurvey(minMass, areaDeg2, zMin, zMax, H0, Om0, Ob0, sigma8, ns, enableDrawSample = True)

selFnCollection=pipelines.makeSelFnCollection(config, mockSurvey)

Expand Down
6 changes: 3 additions & 3 deletions docs/usage.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
.. _UsagePage:

======================
Installation and Usage
======================
=============================
Introduction and Installation
=============================


.. mdinclude:: ../README.md
Expand Down
Loading

0 comments on commit 1fb2192

Please sign in to comment.