Skip to content

Job file structure

Jump to bottom
Chris Churas edited this page Jun 5, 2017 · 21 revisions

This page shows the file and directory structure of a CHM job.

Structure in a tree view with descriptions of files and directories below:

.
|-- base.chm.tasks.list
|-- batched.chm.tasks.list
|-- base.merge.tasks.list
|-- batched.merge.tasks.list
|-- chmrun/
|   |-- mergestdout/
|   |-- overlaymaps/
|   |-- probmaps/
|   |-- stdout/
|   |-- tiles/
|   |   `-- DIRECTORY FOR EACH INPUT IMAGE/
|   `-- tmp/
|-- readme.txt
|-- runjobs.CLUSTER
`-- runmerge.CLUSTER
base.chm.tasks.list
  • Main configuration file for CHM job. Created when createchmjob.py is run and contains CHM tasks to run.
batched.chm.tasks.list
  • Batched CHM task configuration file. This defines how the CHM tasks in base.chm.tasks.list are batched on individual compute nodes in the cluster. Created when checkchmjob.py --submitted is run.
base.merge.tasks.list
  • Configuration of merge tasks. Created when createchmjob.py is run.
batched.merge.tasks.list
  • Batched merge task configuration file. This defines how the merge tasks in base.merge.tasks.list are batched on individual compute nodes in the cluster. Created when checkchmjob.py --submitted is run.
chmrun/
  • Base directory where all job output is written. This directory will always be named this.
chmrun/mergestdout/
  • Directory containing output from merge tasks. Merge tasks are directed to write to this path via runmerge.CLUSTER queue submit script file.
chmrun/overlaymaps/
  • Directory containing output images that are an overlay of the merged probability maps on top of the original input images. These images are created when the merge tasks are run. NOT IMPLEMENTED YET
chmrun/probmaps/
  • Directory containing output merged probability maps. These images are created when the merge tasks are run.
chmrun/stdout/
  • Directory containing output from CHM tasks. CHM tasks are directed to write to this path via runjobs.CLUSTER queue submit script file.
chmrun/tiles/
  • Directory containing directories for each input image. Within each of this input image directories the CHM tasks will write partial probability maps which will be merged by merge tasks in the SECOND phase of processing.
chmrun/tmp/
  • Directory containing temp standard out and standard error files for chmrunner.py and mergetilerunner.py script. Both of these scripts will create a sub directory with format TASKID.UUID/ and under that directory will be a stderr.txt and a stdout.txt containing output from the actual merge and CHM tasks. Once the task completes this directory and files will be removed.
readme.txt
  • Contains information about this job.
runjobs.CLUSTER
  • CHM submit script file where CLUSTER will be set to comet, rocce This file contains necessary flags for submission to the scheduler.
runmerge.CLUSTER
  • Merge submit script file where CLUSTER will be set to comet, rocce This file contains necessary flags for submission to the scheduler.
Clone this wiki locally