Merge pull request #283 from martinkilbinger/canfar_new

Canfar new

docs/wiki/tutorial/pipeline_tutorial.md

@@ -11,7 +11,7 @@
     1. [Path variables](#path-variables)
 1. [Prepare input images](#prepare-input-images)
     1. [Select tiles](#select-tiles)
-    1. [Download tiles and modify names](#download-tiles-and-modify-names)
+    1. [Download tiles](#download-tiles)
     1. [Uncompress tile weights](#uncompress-tile-weights)
 1. [Process single exposure images](#process-single-exposure-images)
     1. [Split images](#split-images)
@@ -67,10 +67,10 @@ Naming and numbering of the input files can closely follow the original image na
   The pixel data can contain the observed image, a weight map, or a flag map. Tile images and weights are created in the
   case of CFIS by Stephen Gwyn using a combination of `swarp` and his own software. Examples of file names are
   `CFIS.316.246.r.fits`, `CFIS.205.267.r.weight.fits.fz`, the latter is a compressed FITS file, see below. Tile flag files
-  are created the mask module of `ShapePipe` (see [Mask images](#mask-images)). The tile ID needs to be modified such that the `.` between the two tile numbers (RA and DEC indicator) is not mistaken for a file extension delimiter. In addition, for
+  are created the mask module of `ShapePipe` (see [Mask images](#mask-images)). The tile ID needs to be modified such that the `.` between the two tile numbers (RA and DEC indicator) is not mistaken for a file extension delimiter. For the same reason, the extension `.fits.fz` is changed to `.fitzfz`. In addition, for
   clarity, we include the string `image` for a tile image type.  
   Default convention: **<image_type>-<tile_number>.fits**  
-  Examples: `CFIS_image-277_282.fits`, `CFIS_weight-274-282.fits.fz`, `pipeline_flag-239_293.fits`
+  Examples: `CFIS_image-277-282.fits`, `CFIS_weight-274-282.fitsfz`, `pipeline_flag-239-293.fits`
 
 - Database catalogue files  
   For very large files that combine information from multiple tiles or single exposures, `ShapePipe` creates `sqlite`
@@ -89,6 +89,9 @@ Naming and numbering of the input files can closely follow the original image na
   The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Select stars](#select-stars)) also outputs ASCII files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM.  
   Example: `star_stat-2366993-18.txt`
 
+- tile numbers list
+  ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)).
+
 - Plots
   The `SETools` module can also produce plots of the objects properties that were selected for a given CCD.
   The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the
@@ -134,8 +137,7 @@ In `$SP_RUN` the following subdirectories need to be created by the user:
 - `input_tiles`: Downloaded (or symbolic links to) CFIS input tile images and weight files.
 - `input_exposures`: Symbolic links to CFIS single-exposure image, weight, and flag files.
 - `output`: General output generated by the pipeline (log files,
-  diagnostics, statistics, output images, catalogues).
-- `output_headers`: Single-exposure headers with WCS information`.
+  diagnostics, statistics, output images, catalogues, Ssingle-exposure headers with WCS information).
 - *Optional*: `output_star_cat`: Star catalogues, only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`.
 
 In general, a call to the pipeline is done as follows, after activating the `shapepipe` conda environment (indicated by `(shapepipe)` at the beginning of the shell prompt.
@@ -154,41 +156,61 @@ Before running `ShapePipe` we need to select and identify tiles and single expos
 The selection of images on input can be done in the config files of the relevant modules, by specifying input
 path(s) and input file name patterns. Either all, or a sub-selection of images in a given input directory can be selected in that way. One might want to pre-select a specific set of images, for example all available images in a given sky area. The resulting files can then be copied to a new, dedicated directory (or alternatively linked using symbolic links), or downloaded to a local machine.
 
-Images can be selected to cover a given sky area, with the script `cfis_field_select`.
+Images can be selected to cover a given sky area, with the script `cfis_field_select`. The tile numbers is all we need here.
 Once we have selected the tiles, we can identify the single exposure images that were used to create those tiles, see [Fine exposures](#find-exposures).
 
-For example, find the tile for a Planck cluster at R.A.=255.66 deg, dec= 34.05 deg can be found with the `--coord` option:
+For example, find the tile number for a Planck cluster at R.A.=255.66 deg, dec= 34.05 deg can be found with the `--coord` option:
 ```bash
-cfis_field_select -i tiles+weights_DR2.txt --coord 255.66deg_34.05deg -v -t tile
+cfis_field_select -i tiles+weights_DR2.txt --coord 255.66deg_34.05deg -t tile --out_name_only --out_number_only -s -o tile_numbers
 ```
-The input text file (with `-i`) contains a list of CFIS tiles.
+The input text file (with `-i`) contains a list of CFIS tiles. The tile number(s) are written as a simple list in the ASCII file `tile_numbers.txt`.
 
-We also need to get the weight files for the tile.
-```bash
-cfis_field_select.py -i tiles+weights_DR2.txt --coord 255.66deg_34.05deg -v -t weight
-```
 
-### Download tiles and modify names
+### Download tiles
 
-The tile images and weights selected in the previous section need to be findable by `ShapePipe` in the tiles input directory `input_tiles`. Either download the images and weights there, or, if they are already stored locally on a hard disk, create symbolic links in `input_tiles`. Now is a good time to make a necessary small change to the file names. As mentioned above, any dot (`.`) that does not indicate a file extension needs to be replaced. In addition, file type specifiers need to appear before the tile number. Therefore, images and weights need to be renamed, for example according to the following scheme:
-```bash
-mv CFIS.424.248.r.fits CFIS_image_424_248.r.fits
-mv CFIS.424.248.r.weight.fits.fz CFIS_weight_424_248.r.weight.fits.fz
+**Module:** get_image_runner  
+**Parent:**  none  
+**Input:** tile numbers list  
+**Output:** tile image, compressed tile weight
+
+The tile images and weights selected in the previous section are made available for `ShapePipe` now, by running the module `get_image_runner`. This module either downloads the images or, if they already exist on a local hard disk, creates symbolic links. Downloading uses the Virtual Observatory VOSpace (vos) software (http://www.ivoa.net/documents/VOSpace). The downloaded files (or link names) are automatically modified to be parsable by the pipeline.
+
+An example config file is `SP_CONFIG/config_get_image.ini`. In the module section, we need to specify
+input path (for example a VOS url), input file pattern and their extension. The input file pattern includes the number(s) as  dummy template, similarly to the `NUMBERING_SCHEME` in the `[FILE]` section (see `File options` in the [general pipeline readme](README.rst)). In addition, the input numbering scheme as python `regexp` needs to be specified, which matches the tile number in the tile number list. 
+
+Next, the output file pattern without the tile number is given.
+
+After that we indicate the copy or download method, for example `vos`. Additional command options can be indicated with the key `COPY_OPTIONS`. Here is the example:
+```ini
+INPUT_PATH = vos:cfis/tiles_DR2, vos:cfis/tiles_DR2
+INPUT_FILE_PATTERN = CFIS.000.000.r, CFIS.000.000.r.weight
+INPUT_FILE_EXT = .fits, .fits.fz
+INPUT_NUMBERING = \d{3}\.\d{3}
+OUTPUT_FILE_PATTERN = CFIS_image-, CFIS_weight-
+COPY = vos
+COPY_OPTIONS = --quick
 ```
-Of course the above renaming can be done at the same time as copying/creating links.
+
+On success, tile images and compressed tile weights are created, either as physical files or symbolic links.
 
 ### Uncompress tile weights
 
-The weights of the stacks are compressed .fits.fz files, they need to be uncompressed before the pipeline is run.
-This can be done with the script `cfis_weight_format`. For example:
+**Module:** uncompress_fits_images_runner  
+**Parent:**  get_image_runner    
+**Input:** compressed tile weight  
+**Output:** tile weight
 
-```bash
-for fzweight in input_tiles/CFIS_weight-*.fits.fz; do
-  weight=`basename $fzweight .fz`
-  cfis_weight_format -i $fzweight -o input_tiles/$weight
-done
+The compressed stack weights (.fits.fz/.fitsfz files) need to be uncompressed before further processing. An example config file is `$SP_CONFIG/config_unfz_w.ini`. Except from the input file pattern and extension (see `File Options` in the [general pipeline readme](README.rst)), we need to specify the output file patern (can be the same as the input), and the HDU number of the weight image data:
+```ini
+[UNCOMPRESS_FITS_IMAGE_RUNNER]
+
+FILE_PATTERN = CFIS_weight
+FILE_EXT = .fitsfz
+OUTPUT_PATTERN = CFIS_weight
+HDU_DATA = 1
 ```
 
+On success, the uncompressed weight image with the correct (only) HDU is written.
 
 ### Find exposures
 
@@ -255,7 +277,7 @@ The example config file is `$SP_CONFIG/config_merge_headers.ini`.
 Specify the output path:
 ```ini
 [MERGE_HEADER_RUNNER]
-OUTPUT_PATH = $SP_RUN/output_headers
+OUTPUT_PATH = $SP_RUN/output
 ```
 Since this produces a single output file instead of a file per input image, it is more convenient to have this file in
 a separated directory for later use.
@@ -567,7 +589,7 @@ DOT_SEX_FILE = $SP_CONFIG/default_tile.sex
 MAKE_POST_PROCESS = TRUE
 
 # Multi-epoch mode: Path to file with single-exposure WCS header information
-LOG_WCS = $SP_Run/output_headers/log_exp_headers.sqlite
+LOG_WCS = $SP_Run/output/log_exp_headers.sqlite
 
 # World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y
 WORLD_POSITION = XWIN_WORLD, YWIN_WORLD
@@ -745,7 +767,7 @@ Here is a commented example config file for the pipeline:
 [NGMIX_RUNNER]
 
 # Create with the split_exp_hdu module
-LOG_WCS = $SP_CONFIG/output_headers/log_exp_headers.npy
+LOG_WCS = $SP_CONFIG/output/log_exp_headers.npy
 ```
 
 ### Create final catalog

environment.yml

@@ -11,9 +11,9 @@ dependencies:
   - pip>=19.3
   - pip:
     - modopt>=1.2
-    - numba==0.40.0
+    - numba>=0.49.1
     - sip_tpv>=1.1
     - sf_tools>=2.0
     - sqlitedict>=1.6.0
     - termcolor>=1.1
-    - git+https://github.com/aguinot/ngmix@modif_observation
+    - git+https://github.com/aguinot/ngmix@stable_version

example/GOLD/config.mask

@@ -51,7 +51,7 @@ MESSIER_MAKE = True
 MESSIER_CAT_PATH = $SP_CONFIG/mask_default/Messier_catalog_updated.npy
 MESSIER_PIXEL_SCALE = 0.187
 MESSIER_SIZE_PLUS = 0.
-MESSIER_FLAG_VALUE = 4
+MESSIER_FLAG_VALUE = 16
 
 
 # -------------------------------- External flag ----------------------------------

example/GOLD/config.mask_tile

@@ -11,9 +11,9 @@ CDSCLIENT_PATH = findgsc2.2
 # ----------------------------- Border parameters ----------------------------------
 [BORDER_PARAMETERS]
 
-BORDER_MAKE = True
+BORDER_MAKE = False
 
-BORDER_WIDTH = 5
+BORDER_WIDTH = 0
 BORDER_FLAG_VALUE = 4
 
 
@@ -51,7 +51,7 @@ MESSIER_MAKE = True
 MESSIER_CAT_PATH = $SP_CONFIG/mask_default/Messier_catalog_updated.npy
 MESSIER_PIXEL_SCALE = 0.187
 MESSIER_SIZE_PLUS = 0.
-MESSIER_FLAG_VALUE = 4
+MESSIER_FLAG_VALUE = 16
 
 
 # -------------------------------- External flag ----------------------------------

example/GOLD/config_exp.ini

@@ -30,13 +30,13 @@ MODE = SMP
 [FILE]
 
 # Log file master name, optional, default: shapepipe
-LOG_NAME = log_sp_exp
+LOG_NAME = log_sp
 
 # Runner log file name, optional, default: shapepipe_runs
-RUN_LOG_NAME = log_run_sp_exp
+RUN_LOG_NAME = log_run_sp
 
 # Input directory, containing input files, single string or list of names with length matching FILE_PATTERN
-INPUT_DIR = $SP_RUN/input_exposures
+INPUT_DIR = last:get_images_runner2
 
 # Output directory
 OUTPUT_DIR = $SP_RUN/output
@@ -87,7 +87,7 @@ FILE_EXT = .fits, .fits, .fits
 NUMBERING_SCHEME = -0000000-0
 
 # Output path (optional)
-OUTPUT_PATH = $SP_RUN/output_headers
+OUTPUT_PATH = $SP_RUN/output
 
 
 [MASK_RUNNER]