Ziggy release 0.7.0 (2024-10-25)

CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use this software in your research, please cite it using these metadata."
 title: Ziggy
-version: v0.6.0
-date-released: "2024-07-26"
+version: v0.7.0
+date-released: "2024-10-25"
 abstract: "Ziggy, a portable, scalable infrastructure for science data processing pipelines, is the child of the Transiting Exoplanet Survey Satellite (TESS) pipeline and the grandchild of the Kepler Pipeline."
 authors:
 - family-names: Tenenbaum

RELEASE-NOTES.md

@@ -4,6 +4,49 @@
 
 These are the release notes for Ziggy. In the change log below, we'll refer to our internal Jira key for our benefit. If the item is associated with a resolved GitHub issue or pull request, we'll add a link to that. Changes that are incompatible with previous versions are marked below. While the major version is 0, we will be making breaking changes when bumping the minor number. However, once we hit version 1.0.0, incompatible changes will only be made when bumping the major number.
 
+# v0.7.0: Halloween release
+
+This release is coming out just before Halloween, and it's full of tricks and treats. Behind the scenes, we continued to buy down decades of technical debt. Are we finally getting close to paying off that loan? By eliminating the StateFile API (ZIGGY-465) and fixing ZIGGY-432, ZIGGY-454, and ZIGGY-478, the pipeline no longer stalls or crashes for mysterious reasons. We fixed a few UI annoyances like collapsing tree controls and Control-Click not working as expected on the Mac.
+
+## New Features
+
+1. Rename sub-task to subtask (ZIGGY-79)
+1. Use log4j2 conventions and features (ZIGGY-82)
+1. Provide means for clients/algorithms to add their software version to Ziggy's data accountability (ZIGGY-430)
+1. Fix Group design (ZIGGY-431)
+1. Refactor PipelineTask (ZIGGY-433)
+1. Limit console to operations (ZIGGY-441)
+1. Add PipelineInstanceIdOperations methods to PipelineInstanceOperations (ZIGGY-445)
+1. Support copying files to task directory without datastore (ZIGGY-446)
+1. Retrieve DatastoreRegexps from the database by name (ZIGGY-447)
+1. Locate consumed files used to produce a file (ZIGGY-448)
+1. Ensure importer can add and update module and pipeline definitions (ZIGGY-452)
+1. Write HDF5 files usable by Zowie (ZIGGY-455)
+1. Eliminate the need for programmatic appenders (ZIGGY-456)
+1. TaskMonitor doesn't change processing step from QUEUED to EXECUTING (ZIGGY-457)
+1. Add parameter retrieval to PipelineTaskOperations (ZIGGY-460)
+1. Check for new vs existing files in datastore (ZIGGY-461)
+1. Eliminate StateFile API (ZIGGY-465)
+
+## Bug Fixes
+
+1. Double-click resize is lost when table auto-update occurs (ZIGGY-297)
+1. Collapsing Parameter Library and Pipelines tree controls (ZIGGY-360)
+1. Can't halt SUBMITTED tasks (ZIGGY-424)
+1. Resume monitoring can't be stopped (ZIGGY-425)
+1. Race condition in pipeline workers (ZIGGY-432)
+1. Ziggy C++ Mex build tools set incorrect install name (ZIGGY-444)
+1. Warning alert clears error alert status (ZIGGY-450)
+1. Python distutils module removed from Python 3.12 (ZIGGY-451)
+1. Local processing crashes sporadically (ZIGGY-454)
+1. ZiggyQuery chunkedIn doesn't work (ZIGGY-462)
+1. Remote execution dialog can't parse numbers with commas (ZIGGY-463)
+1. Parameter API populates empty arrays (ZIGGY-468)
+1. Module parameter sets in HDF5 have incorrect field order values (ZIGGY-469)
+1. Worker never exits when subtask errors (ZIGGY-478)
+1. Control-Click clears selection on the Mac (ZIGGY-479)
+1. Exceptions when using pipeline instance filters (ZIGGY-489)
+
 # v0.6.0: You never have to wonder what Ae 4 / 3 / 0 means again
 
 We fixed a confusing aspect of the user interface and a ton of bugs while we continued to buy down decades of technical debt. You can now halt tasks or instances from the command-line interface (CLI). We improved pipeline definitions by making datastore definitions more flexible and providing for user-specified data receipt unit of work (UOW) generators.

build.gradle

@@ -124,64 +124,6 @@ java {
     withSourcesJar()
 }
 
-test {
-    systemProperty "java.library.path", "$outsideDir/lib"
-    maxHeapSize = "1024m"
-
-    testLogging {
-        events "failed", "skipped"
-    }
-    useJUnit {
-        // If a code coverage report that incudes the integration tests is desired, then comment
-        // out the IntegrationTestCategory line and uncomment the RunByNameTestCategory line. When
-        // the JaCoCo issue described below is resolved, then delete this comment.
-        // excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory'
-        excludeCategories 'gov.nasa.ziggy.IntegrationTestCategory'
-    }
-
-    // Use "gradle -P traceTests test" to show test order.
-    if (project.hasProperty("traceTests")) {
-        afterTest { desc, result ->
-            logger.quiet "${desc.className}.${desc.name}: ${result.resultType}"
-        }
-    }
-}
-
-// Execute tests marked with @Category(IntegrationTestCategory.class).
-task integrationTest(type: Test) {
-    systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml"
-    systemProperty "ziggy.logfile", "$buildDir/build.log"
-    systemProperty "java.library.path", "$outsideDir/lib"
-
-    testLogging {
-        events "failed", "skipped"
-    }
-    useJUnit {
-        includeCategories 'gov.nasa.ziggy.IntegrationTestCategory'
-        excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory'
-    }
-}
-
-// Execute tests marked with @Category(RunByNameTestCategory.class).
-// These tests are typically run explicitly with the --tests option
-// since they don't play well with others. For example:
-// gradle runByNameTests --tests *RmiInterProcessCommunicationTest
-task runByNameTest(type: Test) {
-    systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml"
-    systemProperty "ziggy.logfile", "$buildDir/build.log"
-    systemProperty "java.library.path", "$outsideDir/lib"
-
-    useJUnit {
-        includeCategories 'gov.nasa.ziggy.RunByNameTestCategory'
-    }
-}
-
-// Task specified by the Ziggy Software Management Plan (SMP) to run all tests.
-task testAll
-
-testAll.dependsOn test, integrationTest
-check.dependsOn testAll
-
 // To view code coverage, run the jacocoTestReport task and view the output in:
 // build/reports/jacoco/test/html/index.html.
 check.dependsOn jacocoTestReport
@@ -234,13 +176,18 @@ tasks.withType(com.github.spotbugs.snom.SpotBugsTask) {
 task copyOutsideLibs
 compileJava.dependsOn copyOutsideLibs
 
+
 // Apply Ziggy Gradle script plugins.
+
+// A couple of the other scripts depend on integrationTest.
+apply from: "script-plugins/test.gradle"
+
+// Most scripts in alphabetical order.
 apply from: "script-plugins/copy.gradle"
 apply from: "script-plugins/database-schemas.gradle"
 apply from: "script-plugins/eclipse.gradle"
 apply from: "script-plugins/hdf5.gradle"
 apply from: "script-plugins/misc.gradle"
-apply from: "script-plugins/test.gradle"
 apply from: "script-plugins/wrapper.gradle"
 apply from: "script-plugins/xml-schemas.gradle"
 apply from: "script-plugins/ziggy-libraries.gradle"

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java

@@ -124,7 +124,7 @@ public Set<File> additionalFiles() {
 
     @TaskAction
     public void action() {
-        log.info(String.format("%s.action()\n", this.getClass().getSimpleName()));
+        log.info("{}.action()", this.getClass().getSimpleName());
         File matlabHome = matlabHome();
 
         File buildBinDir = new File(getProject().getBuildDir(), "bin");
@@ -150,9 +150,9 @@ public void action() {
             path += ".app";
             executable = new File(path);
             String message = String.format(
-                "The outputExecutable, \"%s\", already exists and cannot be deleted\n", executable);
+                "The outputExecutable, %s, already exists and cannot be deleted\n", executable);
             if (executable.exists()) {
-                log.info(String.format("%s: already exists, delete it\n", executable));
+                log.info("{} already exists, delete it", executable);
                 if (executable.isDirectory()) {
                     try {
                         FileUtils.deleteDirectory(executable);
@@ -198,7 +198,7 @@ public void action() {
             processBuilder.environment()
                 .put("MCC_DIR", getProject().getProjectDir().getCanonicalPath());
         } catch (IOException e) {
-            log.error(String.format("Could not set MCC_DIR: %s", e.getMessage()), e);
+            log.error("Could not set MCC_DIR: {}", e.getMessage(), e);
         }
         execProcess(processBuilder);
 

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java

@@ -297,7 +297,7 @@ public void setMatlabPath() {
         Project project = getProject();
         if (project.hasProperty(MATLAB_PATH_PROJECT_PROPERTY)) {
             matlabPath = project.findProperty(MATLAB_PATH_PROJECT_PROPERTY).toString();
-            log.info("MATLAB path set from project extra property: " + matlabPath);
+            log.info("MATLAB path set from project extra property {}", matlabPath);
         }
         if (matlabPath == null) {
             String systemPath = System.getenv("PATH");
@@ -307,7 +307,7 @@ public void setMatlabPath() {
                     String pathLower = path.toLowerCase();
                     if (pathLower.contains("matlab") && path.endsWith("bin")) {
                         matlabPath = path.substring(0, path.length() - 4);
-                        log.info("MATLAB path set from PATH environment variable: " + matlabPath);
+                        log.info("MATLAB path set from PATH environment variable {}", matlabPath);
                         break;
                     }
                 }
@@ -317,7 +317,7 @@ public void setMatlabPath() {
             String matlabHome = System.getenv(MATLAB_PATH_ENV_VAR);
             if (matlabHome != null) {
                 matlabPath = matlabHome;
-                log.info("MATLAB path set from MATLAB_HOME environment variable: " + matlabPath);
+                log.info("MATLAB path set from MATLAB_HOME environment variable {}", matlabPath);
             }
         }
         if (matlabPath == null) {

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java

@@ -256,7 +256,7 @@ public String generateMexCommand(File mexfile, File obj) {
     @Override
     public void action() {
 
-        log.info(String.format("%s.action()\n", this.getClass().getSimpleName()));
+        log.info("{}.action()", this.getClass().getSimpleName());
 
         // Start by performing the compilation
         compileAction();

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppPojo.java

@@ -240,8 +240,8 @@ private void populateSourceFiles(boolean warn) {
                 fileListBuilder.append(file.getName());
                 fileListBuilder.append(" ");
             }
-            log.info("List of C/C++ files in directory " + sourceFilePaths + ": "
-                + fileListBuilder.toString());
+            log.info("List of C/C++ files in directory {}: {}", sourceFilePaths,
+                fileListBuilder.toString());
         }
     }
 
@@ -490,7 +490,7 @@ protected DefaultExecutor getDefaultExecutor(File workingDirectory) {
      */
     public void action() {
 
-        log.info(String.format("%s.action()\n", this.getClass().getSimpleName()));
+        log.info("{}.action()", this.getClass().getSimpleName());
 
         // compile the source files
         compileAction();
@@ -505,7 +505,7 @@ protected void compileAction() {
 
         File objDir = objDir();
         if (!objDir.exists()) {
-            log.info("mkdir: " + objDir.getAbsolutePath());
+            log.info("Creating directory {}", objDir.getAbsolutePath());
             objDir.mkdirs();
         }
 
@@ -551,7 +551,7 @@ protected void linkAction() {
             destDir = libDir();
         }
         if (!destDir.exists()) {
-            log.info("mkdir: " + destDir.getAbsolutePath());
+            log.info("Creating directory {}", destDir.getAbsolutePath());
             destDir.mkdirs();
         }
         try {