Skip to content

Latest commit

 

History

History
119 lines (78 loc) · 5.11 KB

README.adoc

File metadata and controls

119 lines (78 loc) · 5.11 KB
Raw

GGCMI2DSSATW

This application is used to extract data from GGCMI NetCDF files and convert that data to DSSAT Weather files.

Features

  • Unit conversion

  • Automatic calculation of monthly temperature averages and amplitude over the entire period of record.

  • Leap year support

  • Specify time window to extract TODO

Usage

Please check the JSON files in the samples directory of this repository for configuration examples.

$ ggcmi2dssatw config.json

This will run according to the config.json file on 1 MPI process.

$ mpiexec -n 27 ggcmi2dssatw config.json

This will run according to the config.json file across 27 MPI processes.

Configuration

All user configuration options are held in a JSON file. For a configuration examples, check the samples directory in the repository.

start_year

The year the NetCDF files start.

end_year

The year the NetCDF files end. TODO

output_dir

The directory to output the DSSAT weather files. NOTE: This directory MUST exists prior to running

extent

A json object consisting of top_left and bottom_right coordinates. These MUST be specified as Longitude/Latitude points. If the points do not align to the GGCMI grid, the closest points which would include the specified bounds will be chosen. TODO: Alignment to GGCMI grid, can be used if MANUALLY aligned to grid

Note
 This mode is exclusive of points mode.
points

A json array of Longitude/Latitude points. These MUST align to the GGCMI grid. TODO

Note
 This mode is exclusive of extent mode.
mapping

The mapping of NetCDF to DSSAT variables, described in Configuration Mapping.

Longitude/Latitude points

Each longitude/latitude point is defined as a JSON array, formatted [longitude,latitude]. They are in decimal degrees, with N/E as positive values and S/W as negative values. Each point SHOULD align with the .5 degree increments set by GGCMI (ending in .25 or .75).

Configuration Mapping

According to GGCMI protocol, each NetCDF file holds the data for one variable. This section handles mapping the NetCDF variable to the equivalent DSSAT variable. Each NetCDF file should defined the following:

file

The file name of the NetCDF file. required

netcdfVar

The variable the NetCDF file represents. required

dssatVar

The equivalent DSSAT variable. required

sourceUnit

The unit of measure of the NetCDF variable. required if mapping units

targetUnit

The unit of mesure of the DSSAT variable. required if mapping units

Warning
 Currently, computing variables from other variables is not supported.

The units are specified according to the udunits2 library.

Execution Modes (TODO)

GGCMI2DSSATW can work in one of two execution modes, extent or points. In extent mode, the user specifies a bounding box of upper left and lower right coordinates and DSSAT weather files will be generated for every point in the bounding box. This is the defaultmode, and if no extent is specified, the entire globe will be run.

In points mode, DSSAT weather files will be generated for user-specified points.

Parallelization using MPI

MPI is used to parallelize the execution of GGCMI2DSSATW. Because of the heavy I/O involved in this process, with many small files possible, it is best practice to assign 1-2 MPI processes per node. Depending on the execution mode, the following algorithm is used to assign jobs to the MPI processes:

Extent Mode

If the # of MPI processes is 4 or less: Divide the extent into x horizontal bands, with any remainders being placed in the last band.

If the # of MPI processes is greater than 4: Divide the extent into 4 horizontal bands, with any overflow being placed into the last band. Divide the extent into x/4 vertical bands, with any remainder bands being added the the first x%4 horizontal bands. Any overflow will be placed in the last veritcal band.

Table 1. Example with 3 processes, 200x200 extent (40000 points)

1 (200x66 = 13200)

2 (200x66 = 13200)

3 (200x68 = 13600)
Table 2. Example with 27 processes, 200x210 extent (420000 points)

1(28x52)

2(28x52)

3(28x52)

4(28x52)

5(28x52)

6(28x52)

7(32x52)

8(28x52)

9(28x52)

10(28x52)

11(28x52)

12(28x52)

13(28x52)

14(32x52)

15(28x52)

16(28x52)

17(28x52)

18(28x52)

19(28x52)

20(28x52)

21(32x52)

22(33x54)

23(33x54)

24(33x54)

25(33x54)

26(33x54)

27(35x54)

 — 

Points Mode (TODO)

If the # of MPI processes is less than # of points: Round-robin allocation of points to processes.

If the # of MPI processes is equal to # of points: Run one point per process.

If the # of MPI processis is greater than # of points: Warn the user of misallocated processes and under utilize the processes.

More Notes about MPI Parallelization

This application has not been designed with memory efficiency in mind. ALL data under every point (including every day), is loaded into memory at the same time. This cuts down on reading I/O, but does make the application consume significantly more memory. The more processors assigned to the MPI job, the less memory each processor needs to accomplish the job.