Building PGE Container Images
The primary delivery artifacts for the OPERA PGE Subsystem consist of per-PGE Docker images which "layer" the opera-sds-pge repository code on top of existing SAS images. This page details the process for building and executing these containers.
Typically, this process is carried out automatically by the Jenkins CI/CD pipeline when a new release is created (see Creating a Release), however, manual creation of images is important for troubleshooting PGE/SAS integration issues, as well as performing demos of PGE release.
If you have not already, consult this page on setting up a development environment for the opera-sds-pge repository. This page assumes that the reader already has a copy of the repository checked out and ready to build.
As mentioned above, the PGE Docker image build process involves layering PGE code and functionality into an existing Docker image containing the corresponding SAS. For OPERA, SAS images are pulled down from the JPL Artifactory-FN so a working user account for the site is required. Contact a PGE developer to request access, but note that access is only available for current JPL employees.
These instructions also assume that all activities are taking place behind the JPL firewall, though VPN or similar means. Attempting to access the Artifactory-FN images will not work otherwise.
Once your Artifactory-FN account is provisioned, we need to configure the local Docker installation to authenticate itself to AF so it can pull down the SAS images on which the PGE images will be based on.
Within your home directory, copy (or append) the following contents to ~/.docker/config.json:
{
"auths": {
"artifactory-fn.jpl.nasa.gov:16001": {
}
}
}
Once added, run docker login artifactory-fn.jpl.nasa.gov:16001, and provide your Artifactory-FN username and password (typically the same as your JPL LDAP credentials). This will authenticate Docker to the Artifactory endpoint for pulling images, and the credentials should become cached locally so docker login does not need to be rerun before every build attempt.
IMPORTANT NOTE: When your jpl password changes the docker login will fail. This is easily remedied by logging with your new password:
docker login artifactory-fn.jpl.nasa.gov:16001
If you see this error: "=> ERROR [internal] load metadata for artifactory-fn.jpl.nasa.gov:16001/gov/nasa/jpl/opera/adt/opera/" , logging out and logging back in should solve the problem.
docker logout artifactory-fn.jpl.nasa.gov:16001
docker login artifactory-fn.jpl.nasa.gov:16001
The Docker build scripts and configuration files are bundled with the opera-sds-pge repository, under the top-level .ci directory. At a high level, this directory contains the following sub-directories:
-
.ci/docker: Contains thedockerfiledefinitions for each PGE. Each file should be named something likeDockerfile_<pge name>for identification. -
.ci/jenkins: Contains theJenkinsfiles for the PGE CI/CD pipelines. These files and their usage is outside the scope of this article. -
.ci/scripts: Contains the shell (bash) scripts that orchestrate building and testing of PGE Docker images. These scripts should always be the starting point for building and testing a PGE image.
Within the .ci/scripts/<pge name> directories, the following executable scripts are available:
-
build_<pge name>.sh: Script for building the Docker image for a single PGE type -
test_<pge name>.sh: Script for running unit tests and code quality checkers from within an existing PGE Docker image -
test_int_<pge name>.sh: Runs the PGE/SAS integration test using expected assets downloaded from S3
All other scripts or files in the directory are used in combination with the scripts listed above, and should typically not be run stand-alone.
For the rest of this article, we will use the build_dswx_hls.sh and test_dsxw_hls.sh scripts (under .ci/scripts/dswx_hls) as our examples for building and testing a PGE image.
To build a single image, we will invoke the build_dswx_hls.sh script on its own. The script has the following help usage (which should be identical across all build scripts):
$ .ci/scripts/build_dswx_hls.sh --help
Usage: build_dswx_hls.sh [-h|--help] [-s|--sas-image <image name>] [-t|--tag <tag>] [-w|--workspace <path>]Where the arguments are:
-
-h|--help: Print the help usage -
-s|--sas-image <image name>: Provide the location of a existing SAS image to base the PGE image on. This location may be within the user's local Docker repository, or point to an existing image on Artifactory-FN. This can be useful when working with locally modified versions of SAS images. By default, the build scripts are configured to use the latest available SAS release corresponding to the PGE in question. -
-t|--tag <tag>: Provide a custom tag value to tag the built image with. By default, the current user's account name is used, appended with-dev, for examplecollinss-dev. -
-w|--workspace <path>: Provide a local path to use as the working directory for building the image. Defaults to the top-level of the local checkout of the opera-sds-pge repository.
Typically, these build scripts can be run without arguments to produce a working image. The following is the example output from building the image (for example to build the dswx_hls image):
$ .ci/scripts/dswx_hls/build_dswx_hls.sh
=====================================
Building DSWx-HLS PGE docker image...
=====================================
WORKSPACE: /Users/collinss/repos/opera_pge
IMAGE: opera_pge/dswx_hls
TAG: collinss-dev
SAS_IMAGE: artifactory-fn.jpl.nasa.gov:16001/gov/nasa/jpl/opera/adt/opera/dswx_hls:interface
[+] Building 42.5s (9/9) FINISHED
=> [internal] load build definition from Dockerfile_dswx_hls 0.0s
=> => transferring dockerfile: 53B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for artifactory-fn.jpl.nasa.gov:16001/gov/nasa/jpl/opera/adt/opera/dswx_hls:interface 1.8s
=> [auth] gov/nasa/jpl/opera/adt/opera/dswx_hls:pull token for artifactory-fn.jpl.nasa.gov:16001 0.0s
=> [internal] load build context 1.2s
=> => transferring context: 9.74MB 1.2s
=> CACHED [1/3] FROM artifactory-fn.jpl.nasa.gov:16001/gov/nasa/jpl/opera/adt/opera/dswx_hls:interface@sha256:439c4bdc6d8ec2563344deaa78453b456764af2701bb0f953767d3 0.0s
=> [2/3] COPY --chown=conda:conda docker_image_staging_mwh4aUWyME /home/conda 0.2s
=> [3/3] RUN set -ex && cd /home/conda && mkdir -p /opt/conda/bin && cp /home/conda/opera/scripts/*_entrypoint.sh /opt/conda/bin && chmod +x /opt/c 36.7s
=> exporting to image 2.4s
=> => exporting layers 2.4s
=> => writing image sha256:59fc61c20497aada0434c639fe1d6577925bf16ce046c21faa1cd9c3e6656731 0.0s
=> => naming to docker.io/opera_pge/dswx_hls:collinss-dev 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
DSWx-HLS PGE Docker image build complete
Cleaning up staging directory /Users/collinss/repos/opera_pge/docker_image_staging_mwh4aUWyME...
You should now see the new image, with the appropriate tag, available within the local Docker repository:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
opera_pge/dswx_hls collinss-dev 59fc61c20497 3 minutes ago 2.2GB
...
The image is now ready for use via the docker run command. Consult the ICS documents for the specific PGE release for further instructions on setting up the appropriate docker run command for the PGE image.
Once the PGE image is build and available from the local Docker repository, we can run test_dswx_hls.sh script to run pytest against the available unit tests from within the image. Running tests this way is important, as some unit tests are either disabled or behave differently when run from outside a containerized context (for example, from within an IDE on the repository itself).
In addition to the unit tests, the test_dswx_hls.sh script will also invoke pylint and flake8 to generate code quality metrics against the PGE code available within the container.
The test script has the following help usage (which again should be consistent across all PGE types):
$ .ci/scripts/test_dswx_hls.sh --help
Usage: test_dswx_hls.sh [-h|--help] [-s|--sas-image <image name>] [-t|--tag <tag>] [-w|--workspace <path>]Where the argument usage is the same as for build_dswx_hls.sh
After running the script, all results will be placed into a test_results/<pge name> directory within the designated workspace. The following is example output from running the test script:
$ .ci/scripts/dswx_hls/test_dswx_hls.sh
=====================================
Testing DSWx-HLS PGE Docker image...
=====================================
Test results output directory: /Users/collinss/repos/opera_pge/test_results/dswx_hls
Unable to create directory /.cache/pylint
Unable to create file /.cache/pylint/opera1.stats: [Errno 2] No such file or directory: '/.cache/pylint/opera1.stats'
DSWx-HLS PGE Docker image test complete
The message about failure to create /.cache/pylint/opera1.stats may be safely ignored. After the script completes, the following files should be available within the test_results/<pge name> directory:
$ ls test_results/dswx_hls/
total 60
drwxrwxr-x 24 collinss 703763885 768 Mar 28 13:53 coverage_html
-rw-rw-r-- 1 collinss 703763885 33940 Apr 15 13:58 flake8.log
-rw-rw-r-- 1 collinss 703763885 10575 Apr 15 13:58 pylint.log
-rw-rw-r-- 1 collinss 703763885 5661 Apr 15 13:58 pytest-junit.xml
-rw-rw-r-- 1 collinss 703763885 2928 Apr 15 13:58 pytest.logThe pytest results are available within pytest.log. To see the current code coverage metrics open coverage_html/index.html within a web browser. From there, you can click on the individual module names to see line-by-line breakdowns of where the unit tests provided coverage or not. For OPERA PGE, we strive to maintain an average of 90% coverage or better across all modules.
flake8.log and pylint.log contain the code quality findings from flake8 and pylint, respectively. Typically, we want to keep the findings within these files to a minimum.
Lastly, pytest-junit.xml is only used by the Jenkins CI pipelines, and may be ignored for the purposes of this article.
This section covers how to integrate Docker image build/test scripts for new PGE additions.
TBD until code refactoring takes place to better accommodate multiple PGE builds