Skip to content

DSWx‐HLS CalVal Acceptance Testing Instructions

Jump to bottom
Scott Collins edited this page Aug 5, 2024 · 1 revision

This page contains instructions for performing Acceptance Testing for the DSWx-HLS Calibration and Validation (CalVal) delivery from the OPERA-ADT team. These instructions pertain to the latest version of the CalVal release, currently v3.3. These instructions assume the user has access to the JPL FN-Artifactory, and has Docker installed on their local machine.

Acquiring the DSWx-HLS CalVal Docker Image

The image is currently hosted on JPL FN-Artifactory, which requires JPL VPN access and JPL credentials. You may also need to be added to the gov.nasa.jpl.opera.adt organization.

Once you have access, the container tarball delivery is available under general/gov/nasa/jpl/opera/adt/dswx_hls/r3.3/cal_val/dockerimg_proteus_cal_val_3.3.tar. Sample inputs and outputs are also available under general/gov/nasa/jpl/opera/adt/dswx_hls/r3.3/cal_val/.

Download both images to a location on your local machine. This location will be referred to throughout this instructions as <DSWx_DIR>

Loading the image into Docker

The first step in running the DSWx-HLS image is to load it into Docker via the following command:

docker load -i <DSWx_DIR>/dockerimg_proteus_cal_val_3.3.tar

This should add the Docker image to your local repository with the name opera/proteus and the tag cal_val_3.3.

Preparing the test data

Once the delivery_cal_val_3.3.zip file is downloaded to your local machine, unpack it to <DSWx_DIR>:

unzip delivery_cal_val_3.3.zip

This will create a delivery_3_cal_val_point_release_jan_2023 directory within <DSWx_DIR> containing two test datasets, l30_greenland and s30_louisiana. Within each dataset are the following files/directories:

  • expected_output_dir/
  • input_dir/
  • output_dir/
  • dswx_hls.yaml
  • README.txt

In order to execute the SAS, the input file directory, runconfig and an output location will be mounted into container instance as Docker Volumes. To help streamline this process, we recommend making the following changes to the delivery_3_cal_val_point_release_jan_2023 directory:

  1. Rename the delivery_3_cal_val_point_release_jan_2023 directory to test_datasets

    mv <DSWx_DIR>/delivery_3_cal_val_point_release_jan_2023 <DSWx_DIR>/test_datasets

  2. Create a directory named runconfig within each dataset directory, and move the existing runconfig YAML file into them:

    mkdir -p <DSWx_DIR>/test_datasets/{l30_greenland|s30_louisiana}/runconfig

    mv <DSWx_DIR>/test_datasets/{l30_greenland|s30_louisiana}/dswx_hls.yaml <DSWx_DIR>/test_datasets/{l30_greenland|s30_louisiana}/runconfig

Executing the DSWx-HLS container on the sample datasets

We're now ready to execute the DSWx-HLS CalVal release. Run the following the command to kick off execution for each dataset:

docker run --rm -u $UID:$(id -g) \
  -v <DSWx_DIR>/test_datasets/<DATASET>/runconfig:/home/conda/runconfig:ro \
  -v <DSWx_DIR>/test_datasets/<DATASET>/input_dir:/home/conda/input_dir:ro \
  -v <DSWx_DIR>/test_datasets/<DATASET>/output_dir:/home/conda/output_dir \
  -i --tty opera/proteus:cal_val_3.3 \
  sh -ci "python3 proteus-0.5.2/bin/dswx_hls.py runconfig/dswx_hls.yaml --log output_dir/<DATASET>.log"

Where <DATASET> is one of l30_greenland or s30_louisiana.

You should see console output from the executing SAS container. Execution should only take roughly 10-20 minutes per dataset. Once execution is complete, you should see a number of dswx_hls_v0.5.2_B##_<BAND>.tif files, as well as the dswx_hls_v0.5.2_BROWSE.tif, dswx_hls_v0.5.2_BROWSE.png and <DATASET>.log files.

Running the Quality Assurance test

Now that we've successfully executed the SAS container and generated outputs, the last step is to perform a QA check against the expected outputs.

The first step is to create a script that will compute difference images between the output and expected TIF files using the dswx_compare.py utility included with the SAS container delivery. Within the <DSWx_DIR> directory, create a script named dswx_compare_results.sh with the following contents:

#!/bin/bash

WORKING_DIR=`pwd`
DATASET=$1

declare -a dswx_bands=( "dswx_hls_v0.5.2_B01_WTR.tif"
                        "dswx_hls_v0.5.2_B02_BWTR.tif"
                        "dswx_hls_v0.5.2_B03_CONF.tif"
                        "dswx_hls_v0.5.2_B04_DIAG.tif"
                        "dswx_hls_v0.5.2_B05_WTR-1.tif"
                        "dswx_hls_v0.5.2_B06_WTR-2.tif"
                        "dswx_hls_v0.5.2_B08_SHAD.tif"
                        "dswx_hls_v0.5.2_B09_CLOUD.tif"
                        "dswx_hls_v0.5.2_B10_DEM.tif"
                        "dswx_hls_v0.5.2_BROWSE.tif" )

for dswx_band in "${dswx_bands[@]}"; do
    echo "Comparing results for $DATASET:$dswx_band"
    docker run --rm -u $UID:$(id -g) \
      -v $WORKING_DIR/$DATASET/output_dir:/home/conda/output_dir:ro \
      -v $WORKING_DIR/$DATASET/expected_output_dir:/home/conda/expected_output_dir:ro \
      -i --tty opera/proteus:cal_val_3.3 \
      sh -ci "python3 proteus-0.5.2/bin/dswx_compare.py output_dir/$dswx_band expected_output_dir/$dswx_band"
done

Make sure the script has execute permissions:

chmod 775 <DSWx_DIR>/dswx_compare_results.sh

Now execute the QA script on each dataset from within <DSWx_DIR>:

./dswx_compare_results.sh l30_greenland
./dswx_compare_results.sh s30_louisiana

The QA console output should appear on screen, with comparisons made for each output tif band:

Comparing results for l30_greenland:dswx_hls_v0.5.2_B01_WTR.tif
Comparing files:
    file 1: output_dir/dswx_hls_v0.5.2_B01_WTR.tif
    file 2: expected_output_dir/dswx_hls_v0.5.2_B01_WTR.tif
[OK]   Comparing number of bands
Comparing DSWx bands...
[OK]        Band 1 - Water classification (WTR)"
[OK]   Comparing geotransform
[OK]   Comparing metadata
...
Comparing results for s30_louisiana:dswx_hls_v0.5.2_B01_WTR.tif
Comparing files:
    file 1: output_dir/dswx_hls_v0.5.2_B01_WTR.tif
    file 2: expected_output_dir/dswx_hls_v0.5.2_B01_WTR.tif
[OK]   Comparing number of bands
Comparing DSWx bands...
[OK]        Band 1 - Water classification (WTR)"
[OK]   Comparing geotransform
[OK]   Comparing metadata
...

The expected results for this Acceptance Test are for all bands to return values of [OK] for all tests.

Clone this wiki locally