Inferring dark matter substructure with machine learning

Code repository for the paper Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning by Johann Brehmer, Siddharth Mishra-Sharma, Joeri Hermans, Gilles Louppe, and Kyle Cranmer.

Abstract

The subtle and unique imprint of dark matter substructure on extended arcs in strong lensing systems contains a wealth of information about the properties and distribution of dark matter on small scales and, consequently, about the underlying particle physics. However, teasing out this effect poses a significant challenge since the likelihood function for realistic simulations of population-level parameters is intractable. We apply recently-developed simulation-based inference techniques to the problem of substructure inference in galaxy-galaxy strong lenses. By leveraging additional information extracted from the simulator, neural networks are efficiently trained to estimate likelihood ratios associated with population-level parameters characterizing substructure. Through proof-of-principle application to simulated data, we show that these methods can provide an efficient and principled way to simultaneously analyze an ensemble of strong lenses, and can be used to mine the large sample of lensing images deliverable by near-future surveys for signatures of dark matter substructure.

Results

In figures/ we collect the figures shown in the paper. The folder also contains a few additional plots and animations:

• deflection_maps.pdf shows an example simulated lens including the subhalos, the deflection map from the host halo, and the deflection map from all subhalos combined.
• calibration_curves.pdf shows calibration curves (raw network output vs calibrated likelihood ratio) for several randomly chosen parameter points.
• live_inference_population.gif shows the evolution of the posterior on the population-level parameters with successive simulated observations.
• live_inference_shmf.gif shows the evolution of the posterior on the subhalo mass function with successive simulated observations.
• live_inference_both.gif and live_inference_with_images.gif combine the evolution of both posteriors, the latter (shown at the top of this readme) also includes the simulated observed lensed images.

Code

The dependencies of our code are listed in environments.yml.

The starting point of our code based are five high-level scripts:

• simulate.py starts the simulation of lensing systems and, depending on the command-line parameters, creates training, calibration, or test samples.
• combine_samples.py makes it easy to combine multiple simulation runs into a single file as a preparation for training.
• train.py trains neural networks on the simulated data.
• test.py steers the evaluation of the estimated likelihood ratio on test or calibration data.
• calibrate.py calibrated network predictions based on histograms of the network output.

All scripts can be called with the argument --help to show the command line options. In scripts/ we collect the actual calls we used (on a HPC environment) during this project.

Generally, the simulation code resides in simulation, while the inference code is in the inference folder. Notebooks in notebooks contain the plotting code.

Note: Internally the code uses a different convention for the SHMF slope beta than in the paper. The relation is beta_code = beta_paper - 1, so the fiducial value beta = -0.9 from the paper is internally represented as beta = -1.9.

Singularity

To use Singularity, first create the container:

sudo singularity build [name].sif Singularity

To run the code inside the singularity container [name].sif do:

singularity run \
    --nv \ # to enable gpu calculations
    --cleanenv \ # do not transfer any local environment variables
    -B [location_for_large_disk_usage]:/code/data \ # e.g. where to store results, data, etc
    -B tmp:/tmp \ # in case of heavy use of /tmp
    --no-home \ # turn off mount the current directory to home
    --contain \ # turn off mount the current directory to home
    --writable-tmpfs \ # make container temporarily mutable (with very limited space)
    [name].sif \
    python3 test.py -h # or any other command you'd might want to run

Docker

Build Docker images (may require sudo):

docker build --rm -t [docker_name] --no-cache [path_to_dockerfile]

To run experiments do:

docker run \
  --rm \ # remove container upon termination
  -v [location_for_large_disk_usage]:/code/data \ # e.g. where to store results, data, etc
  -w /code \ # working directory inside container
  --gpus all \ # access all gpus (see below for requirements)
  [name_docker_image] \ 
  python3 test.py -h # or any other command you'd might want to run

Docker gpu support

Docker gpu support requires nvidia-container-toolkit and cuda >= 10.2. To install the toolkit, run:

sudo apt update
sudo apt install -y nvidia-container-toolkit

References

If you use this code, please cite our paper:

@article{Brehmer:2019jyt,
      author         = "Brehmer, Johann and Mishra-Sharma, Siddharth and Hermans,
                        Joeri and Louppe, Gilles and Cranmer, Kyle",
      title          = "{Mining for Dark Matter Substructure: Inferring subhalo
                        population properties from strong lenses with machine
                        learning}",
      year           = "2019",
      eprint         = "1909.02005",
      archivePrefix  = "arXiv",
      primaryClass   = "astro-ph.CO",
      SLACcitation   = "%%CITATION = ARXIV:1909.02005;%%"
}

The inference method is based on 1805.00013, 1805.00020, 1805.12244, and 1808.00973.