diff --git a/CITATION.cff b/CITATION.cff index daad847..83a4838 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -1,8 +1,8 @@ cff-version: 1.2.0 message: "If you use this software in your research, please cite it using these metadata." title: Ziggy -version: v0.6.0 -date-released: "2024-07-26" +version: v0.7.0 +date-released: "2024-10-25" abstract: "Ziggy, a portable, scalable infrastructure for science data processing pipelines, is the child of the Transiting Exoplanet Survey Satellite (TESS) pipeline and the grandchild of the Kepler Pipeline." authors: - family-names: Tenenbaum diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index 0b36c04..d0b3b67 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -4,6 +4,49 @@ These are the release notes for Ziggy. In the change log below, we'll refer to our internal Jira key for our benefit. If the item is associated with a resolved GitHub issue or pull request, we'll add a link to that. Changes that are incompatible with previous versions are marked below. While the major version is 0, we will be making breaking changes when bumping the minor number. However, once we hit version 1.0.0, incompatible changes will only be made when bumping the major number. +# v0.7.0: Halloween release + +This release is coming out just before Halloween, and it's full of tricks and treats. Behind the scenes, we continued to buy down decades of technical debt. Are we finally getting close to paying off that loan? By eliminating the StateFile API (ZIGGY-465) and fixing ZIGGY-432, ZIGGY-454, and ZIGGY-478, the pipeline no longer stalls or crashes for mysterious reasons. We fixed a few UI annoyances like collapsing tree controls and Control-Click not working as expected on the Mac. + +## New Features + +1. Rename sub-task to subtask (ZIGGY-79) +1. Use log4j2 conventions and features (ZIGGY-82) +1. Provide means for clients/algorithms to add their software version to Ziggy's data accountability (ZIGGY-430) +1. Fix Group design (ZIGGY-431) +1. Refactor PipelineTask (ZIGGY-433) +1. Limit console to operations (ZIGGY-441) +1. Add PipelineInstanceIdOperations methods to PipelineInstanceOperations (ZIGGY-445) +1. Support copying files to task directory without datastore (ZIGGY-446) +1. Retrieve DatastoreRegexps from the database by name (ZIGGY-447) +1. Locate consumed files used to produce a file (ZIGGY-448) +1. Ensure importer can add and update module and pipeline definitions (ZIGGY-452) +1. Write HDF5 files usable by Zowie (ZIGGY-455) +1. Eliminate the need for programmatic appenders (ZIGGY-456) +1. TaskMonitor doesn't change processing step from QUEUED to EXECUTING (ZIGGY-457) +1. Add parameter retrieval to PipelineTaskOperations (ZIGGY-460) +1. Check for new vs existing files in datastore (ZIGGY-461) +1. Eliminate StateFile API (ZIGGY-465) + +## Bug Fixes + +1. Double-click resize is lost when table auto-update occurs (ZIGGY-297) +1. Collapsing Parameter Library and Pipelines tree controls (ZIGGY-360) +1. Can't halt SUBMITTED tasks (ZIGGY-424) +1. Resume monitoring can't be stopped (ZIGGY-425) +1. Race condition in pipeline workers (ZIGGY-432) +1. Ziggy C++ Mex build tools set incorrect install name (ZIGGY-444) +1. Warning alert clears error alert status (ZIGGY-450) +1. Python distutils module removed from Python 3.12 (ZIGGY-451) +1. Local processing crashes sporadically (ZIGGY-454) +1. ZiggyQuery chunkedIn doesn't work (ZIGGY-462) +1. Remote execution dialog can't parse numbers with commas (ZIGGY-463) +1. Parameter API populates empty arrays (ZIGGY-468) +1. Module parameter sets in HDF5 have incorrect field order values (ZIGGY-469) +1. Worker never exits when subtask errors (ZIGGY-478) +1. Control-Click clears selection on the Mac (ZIGGY-479) +1. Exceptions when using pipeline instance filters (ZIGGY-489) + # v0.6.0: You never have to wonder what Ae 4 / 3 / 0 means again We fixed a confusing aspect of the user interface and a ton of bugs while we continued to buy down decades of technical debt. You can now halt tasks or instances from the command-line interface (CLI). We improved pipeline definitions by making datastore definitions more flexible and providing for user-specified data receipt unit of work (UOW) generators. diff --git a/build.gradle b/build.gradle index d0ffcc4..0061ac3 100644 --- a/build.gradle +++ b/build.gradle @@ -124,64 +124,6 @@ java { withSourcesJar() } -test { - systemProperty "java.library.path", "$outsideDir/lib" - maxHeapSize = "1024m" - - testLogging { - events "failed", "skipped" - } - useJUnit { - // If a code coverage report that incudes the integration tests is desired, then comment - // out the IntegrationTestCategory line and uncomment the RunByNameTestCategory line. When - // the JaCoCo issue described below is resolved, then delete this comment. - // excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory' - excludeCategories 'gov.nasa.ziggy.IntegrationTestCategory' - } - - // Use "gradle -P traceTests test" to show test order. - if (project.hasProperty("traceTests")) { - afterTest { desc, result -> - logger.quiet "${desc.className}.${desc.name}: ${result.resultType}" - } - } -} - -// Execute tests marked with @Category(IntegrationTestCategory.class). -task integrationTest(type: Test) { - systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml" - systemProperty "ziggy.logfile", "$buildDir/build.log" - systemProperty "java.library.path", "$outsideDir/lib" - - testLogging { - events "failed", "skipped" - } - useJUnit { - includeCategories 'gov.nasa.ziggy.IntegrationTestCategory' - excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory' - } -} - -// Execute tests marked with @Category(RunByNameTestCategory.class). -// These tests are typically run explicitly with the --tests option -// since they don't play well with others. For example: -// gradle runByNameTests --tests *RmiInterProcessCommunicationTest -task runByNameTest(type: Test) { - systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml" - systemProperty "ziggy.logfile", "$buildDir/build.log" - systemProperty "java.library.path", "$outsideDir/lib" - - useJUnit { - includeCategories 'gov.nasa.ziggy.RunByNameTestCategory' - } -} - -// Task specified by the Ziggy Software Management Plan (SMP) to run all tests. -task testAll - -testAll.dependsOn test, integrationTest -check.dependsOn testAll - // To view code coverage, run the jacocoTestReport task and view the output in: // build/reports/jacoco/test/html/index.html. check.dependsOn jacocoTestReport @@ -234,13 +176,18 @@ tasks.withType(com.github.spotbugs.snom.SpotBugsTask) { task copyOutsideLibs compileJava.dependsOn copyOutsideLibs + // Apply Ziggy Gradle script plugins. + +// A couple of the other scripts depend on integrationTest. +apply from: "script-plugins/test.gradle" + +// Most scripts in alphabetical order. apply from: "script-plugins/copy.gradle" apply from: "script-plugins/database-schemas.gradle" apply from: "script-plugins/eclipse.gradle" apply from: "script-plugins/hdf5.gradle" apply from: "script-plugins/misc.gradle" -apply from: "script-plugins/test.gradle" apply from: "script-plugins/wrapper.gradle" apply from: "script-plugins/xml-schemas.gradle" apply from: "script-plugins/ziggy-libraries.gradle" diff --git a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java index d07453a..cbf0a41 100644 --- a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java +++ b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java @@ -124,7 +124,7 @@ public Set additionalFiles() { @TaskAction public void action() { - log.info(String.format("%s.action()\n", this.getClass().getSimpleName())); + log.info("{}.action()", this.getClass().getSimpleName()); File matlabHome = matlabHome(); File buildBinDir = new File(getProject().getBuildDir(), "bin"); @@ -150,9 +150,9 @@ public void action() { path += ".app"; executable = new File(path); String message = String.format( - "The outputExecutable, \"%s\", already exists and cannot be deleted\n", executable); + "The outputExecutable, %s, already exists and cannot be deleted\n", executable); if (executable.exists()) { - log.info(String.format("%s: already exists, delete it\n", executable)); + log.info("{} already exists, delete it", executable); if (executable.isDirectory()) { try { FileUtils.deleteDirectory(executable); @@ -198,7 +198,7 @@ public void action() { processBuilder.environment() .put("MCC_DIR", getProject().getProjectDir().getCanonicalPath()); } catch (IOException e) { - log.error(String.format("Could not set MCC_DIR: %s", e.getMessage()), e); + log.error("Could not set MCC_DIR: {}", e.getMessage(), e); } execProcess(processBuilder); diff --git a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java index 97c9c9b..975fb7e 100644 --- a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java +++ b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java @@ -297,7 +297,7 @@ public void setMatlabPath() { Project project = getProject(); if (project.hasProperty(MATLAB_PATH_PROJECT_PROPERTY)) { matlabPath = project.findProperty(MATLAB_PATH_PROJECT_PROPERTY).toString(); - log.info("MATLAB path set from project extra property: " + matlabPath); + log.info("MATLAB path set from project extra property {}", matlabPath); } if (matlabPath == null) { String systemPath = System.getenv("PATH"); @@ -307,7 +307,7 @@ public void setMatlabPath() { String pathLower = path.toLowerCase(); if (pathLower.contains("matlab") && path.endsWith("bin")) { matlabPath = path.substring(0, path.length() - 4); - log.info("MATLAB path set from PATH environment variable: " + matlabPath); + log.info("MATLAB path set from PATH environment variable {}", matlabPath); break; } } @@ -317,7 +317,7 @@ public void setMatlabPath() { String matlabHome = System.getenv(MATLAB_PATH_ENV_VAR); if (matlabHome != null) { matlabPath = matlabHome; - log.info("MATLAB path set from MATLAB_HOME environment variable: " + matlabPath); + log.info("MATLAB path set from MATLAB_HOME environment variable {}", matlabPath); } } if (matlabPath == null) { diff --git a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java index 7f5a3ec..dbc21ae 100644 --- a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java +++ b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java @@ -256,7 +256,7 @@ public String generateMexCommand(File mexfile, File obj) { @Override public void action() { - log.info(String.format("%s.action()\n", this.getClass().getSimpleName())); + log.info("{}.action()", this.getClass().getSimpleName()); // Start by performing the compilation compileAction(); diff --git a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppPojo.java b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppPojo.java index 02b27d2..8d7ed8b 100644 --- a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppPojo.java +++ b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppPojo.java @@ -240,8 +240,8 @@ private void populateSourceFiles(boolean warn) { fileListBuilder.append(file.getName()); fileListBuilder.append(" "); } - log.info("List of C/C++ files in directory " + sourceFilePaths + ": " - + fileListBuilder.toString()); + log.info("List of C/C++ files in directory {}: {}", sourceFilePaths, + fileListBuilder.toString()); } } @@ -490,7 +490,7 @@ protected DefaultExecutor getDefaultExecutor(File workingDirectory) { */ public void action() { - log.info(String.format("%s.action()\n", this.getClass().getSimpleName())); + log.info("{}.action()", this.getClass().getSimpleName()); // compile the source files compileAction(); @@ -505,7 +505,7 @@ protected void compileAction() { File objDir = objDir(); if (!objDir.exists()) { - log.info("mkdir: " + objDir.getAbsolutePath()); + log.info("Creating directory {}", objDir.getAbsolutePath()); objDir.mkdirs(); } @@ -551,7 +551,7 @@ protected void linkAction() { destDir = libDir(); } if (!destDir.exists()) { - log.info("mkdir: " + destDir.getAbsolutePath()); + log.info("Creating directory {}", destDir.getAbsolutePath()); destDir.mkdirs(); } try { diff --git a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyVersionGenerator.java b/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyVersionGenerator.java deleted file mode 100644 index ba3e820..0000000 --- a/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyVersionGenerator.java +++ /dev/null @@ -1,159 +0,0 @@ -package gov.nasa.ziggy.buildutil; - -import java.io.BufferedReader; -import java.io.BufferedWriter; -import java.io.File; -import java.io.FileWriter; -import java.io.IOException; -import java.io.InputStreamReader; -import java.util.ArrayList; -import java.util.List; -import java.util.stream.Collectors; - -import org.gradle.api.DefaultTask; -import org.gradle.api.tasks.Input; -import org.gradle.api.tasks.OutputFile; -import org.gradle.api.tasks.TaskAction; - -import com.google.common.collect.ImmutableList; - -/** - * Places version info in the property file {@value #BUILD_CONFIGURATION}). The available Gradle - * properties used to set the names of the various properties that describe the version include: - *
- *
{@code versionPropertyName}
- *
name of the property that holds the output of {@code git describe} (default: - * {@value #DEFAULT_BUILD_VERSION_PROPERTY_NAME})
- *
{@code branchPropertyName}
- *
name of the property that contains the name of the current branch - * (default:{@value #DEFAULT_BUILD_BRANCH_PROPERTY_NAME})
- *
{@code commitPropertyName}
- *
name of the property that contains the commit - * (default:{@value #DEFAULT_BUILD_COMMIT_PROPERTY_NAME})
- *
- *

- * The following example shows how to to run this plugin only when necessary by saving the current - * Git version in a property called {@code ziggyVersion}. The default values of the aforementioned - * properties are used. - * - * 

- * import gov.nasa.ziggy.buildutil.ZiggyVersionGenerator
- *
- * def gitVersion = new ByteArrayOutputStream()
- * exec {
- *     commandLine "git", "rev-parse", "HEAD"
- *     standardOutput = gitVersion
- * }
- * gitVersion = gitVersion.toString().trim()
- *
- * task ziggyVersion(type: ZiggyVersionGenerator) {
- *     inputs.property "ziggyVersion", gitVersion
- * }
- *
- * - * See ZiggyConfiguration. - */ -public class ZiggyVersionGenerator extends DefaultTask { - - private static final String BUILD_CONFIGURATION = "src/main/resources/ziggy-build.properties"; - private static final String DEFAULT_BUILD_VERSION_PROPERTY_NAME = "ziggy.version"; - private static final String DEFAULT_BUILD_BRANCH_PROPERTY_NAME = "ziggy.version.branch"; - private static final String DEFAULT_BUILD_COMMIT_PROPERTY_NAME = "ziggy.version.commit"; - - /** - * Length that object names are abbreviated to. The number is set to 10, because that is - * currently 1 more than necessary to distinguish all commit hashes to date. One command to - * determine the maximum hash length in use is: - * - * 
-     * $ git rev-list --all --abbrev=0 --abbrev-commit | awk '{print length()}' | sort -n | uniq -c
-     * 1040 4
-     * 7000 5
-     * 1149 6
-     *   68 7
-     *    8 8
-     *    1 9
-     *
- */ - private static final int ABBREV = 10; - - private static final String HEADER = "# This file is automatically generated by Gradle." - + System.lineSeparator(); - - private String versionPropertyName = DEFAULT_BUILD_VERSION_PROPERTY_NAME; - private String branchPropertyName = DEFAULT_BUILD_BRANCH_PROPERTY_NAME; - private String commitPropertyName = DEFAULT_BUILD_COMMIT_PROPERTY_NAME; - - @TaskAction - public void generateVersionProperties() throws IOException, InterruptedException { - - File outputFile = new File(getProject().getProjectDir(), getBuildConfiguration()); - - try (BufferedWriter output = new BufferedWriter(new FileWriter(outputFile))) { - output.write(HEADER); - output.write(getVersionPropertyName() + " = " - + runCommand(ImmutableList.of("git", "describe", "--always", "--abbrev=" + ABBREV)) - + System.lineSeparator()); - output.write(getBranchPropertyName() + " = " - + runCommand(ImmutableList.of("git", "rev-parse", "--abbrev-ref", "HEAD")) - + System.lineSeparator()); - output.write(getCommitPropertyName() + " = " - + runCommand(ImmutableList.of("git", "rev-parse", "--short=" + ABBREV, "HEAD")) - + System.lineSeparator()); - } - } - - public String runCommand(List command) throws IOException, InterruptedException { - - Process process = new ProcessBuilder(command).start(); - List output = new ArrayList<>(); - - BufferedReader bufferedReader = new BufferedReader( - new InputStreamReader(process.getInputStream())); - - for (;;) { - String line = bufferedReader.readLine(); - if (line == null) { - break; - } - output.add(line); - } - - process.waitFor(); - - return output.stream().collect(Collectors.joining(System.lineSeparator())).trim(); - } - - /** Override this to create your own subclass for pipeline-side version generation. */ - @OutputFile - public String getBuildConfiguration() { - return BUILD_CONFIGURATION; - } - - @Input - public String getVersionPropertyName() { - return versionPropertyName; - } - - public void setVersionPropertyName(String versionPropertyName) { - this.versionPropertyName = versionPropertyName; - } - - @Input - public String getBranchPropertyName() { - return branchPropertyName; - } - - public void setBranchPropertyName(String branchPropertyName) { - this.branchPropertyName = branchPropertyName; - } - - @Input - public String getCommitPropertyName() { - return commitPropertyName; - } - - public void setCommitPropertyName(String commitPropertyName) { - this.commitPropertyName = commitPropertyName; - } -} diff --git a/doc/build.gradle b/doc/build.gradle deleted file mode 100644 index 0933172..0000000 --- a/doc/build.gradle +++ /dev/null @@ -1,115 +0,0 @@ -defaultTasks 'build' - -def getDate() { - def date = new Date() - def formattedDate = date.format('yyyyMMdd') - return formattedDate -} - -task cleanestDryRun(type: Exec) { - description = "Removes pdf and .gradle directories (DRY RUN)." - - outputs.upToDateWhen { false } - - workingDir = rootDir - commandLine "sh", "-c", "git clean --force -x -d --dry-run" -} - -task cleanest(type: Exec) { - description = "Removes pdf and .gradle directories." - - outputs.upToDateWhen { false } - - workingDir = rootDir - commandLine "sh", "-c", "git clean --force -x -d" -} - -subprojects { - defaultTasks 'build' - - task build() { - } - - task makeDocId() { - description = "Generates a doc-id.sty file." - - inputs.files fileTree(dir: projectDir, include: '**/*.tex', exclude: '**/build/**').files - outputs.file "$buildDir/doc-id.sty" - - makeDocId.doFirst { - mkdir buildDir - } - - doLast() { - if (!project.hasProperty('docId')) { - return - } - - exec { - workingDir buildDir - commandLine "bash", "-c", "echo -E '\\newcommand{\\DOCID}{$docId}' > doc-id.sty" - } - } - } - - task compileLaTex(dependsOn: makeDocId) { - description = "Compiles the .tex files into a .pdf file." - - inputs.files fileTree(dir: projectDir, include: '**/*.tex', exclude: '**/build/**').files - outputs.files fileTree(dir: buildDir, include: '**/*.pdf').files - - doFirst { - mkdir buildDir - } - - doLast { - if (!project.hasProperty('texFileName')) { - return - } - - // Execute twice to update references and a third time for BibTeX. - 3.times { - exec { - executable 'pdflatex' - workingDir project.workingDir - args '-output-directory=build' - args '-interaction=nonstopmode' - args '-halt-on-error' - args texFileName - } - } - } - } - build.dependsOn compileLaTex - - task publish(dependsOn: build) { - description = "Publishes the .pdf file into the pdf directory." - - inputs.dir buildDir - outputs.files fileTree(rootDir.getPath() + '/pdf').include('**/*-' + getDate() + '.pdf').files - - doFirst() { - mkdir rootDir.getPath() + '/pdf/' + publishDir - } - - doLast() { - if (!project.hasProperty('texFileName') || !project.hasProperty('publishDir') || !project.hasProperty('docId')) { - return - } - - copy { - from(buildDir) { - rename '^(.*).pdf$', docId + '-$1-' + getDate() + '.pdf' - } - into rootDir.getPath() + '/pdf/' + publishDir - include '**/*.pdf' - } - } - } - - task clean() { - doLast() { - delete buildDir - } - } -} diff --git a/doc/user-manual/advanced-uow.md b/doc/user-manual/advanced-uow.md index a416bc5..4526300 100644 --- a/doc/user-manual/advanced-uow.md +++ b/doc/user-manual/advanced-uow.md @@ -2,7 +2,7 @@ [[Previous]](nicknames.md) [[Up]](dusty-corners.md) -[[Next]](contact-us.md) +[[Next]](version-tracking.md) ## Advanced Unit of Work Configurations @@ -69,4 +69,4 @@ Fortunately, the `year` and `doy` parts of the `fileNameRegexp` will respect the [[Previous]](nicknames.md) [[Up]](dusty-corners.md) -[[Next]](contact-us.md) +[[Next]](version-tracking.md) diff --git a/doc/user-manual/images/edit-array-1.png b/doc/user-manual/images/edit-array-1.png index 8b2a917..ab15c27 100644 Binary files a/doc/user-manual/images/edit-array-1.png and b/doc/user-manual/images/edit-array-1.png differ diff --git a/doc/user-manual/images/edit-param-set.png b/doc/user-manual/images/edit-param-set.png index 4920640..94c74a9 100644 Binary files a/doc/user-manual/images/edit-param-set.png and b/doc/user-manual/images/edit-param-set.png differ diff --git a/doc/user-manual/images/param-lib-context-menu.png b/doc/user-manual/images/param-lib-context-menu.png index a5b71f6..37e83d3 100644 Binary files a/doc/user-manual/images/param-lib-context-menu.png and b/doc/user-manual/images/param-lib-context-menu.png differ diff --git a/doc/user-manual/images/pipelines-config-1.png b/doc/user-manual/images/pipelines-config-1.png index 5be97c0..f954dde 100644 Binary files a/doc/user-manual/images/pipelines-config-1.png and b/doc/user-manual/images/pipelines-config-1.png differ diff --git a/doc/user-manual/images/pipelines-config-2.png b/doc/user-manual/images/pipelines-config-2.png index d9d332a..a52bab2 100644 Binary files a/doc/user-manual/images/pipelines-config-2.png and b/doc/user-manual/images/pipelines-config-2.png differ diff --git a/doc/user-manual/rdbms.md b/doc/user-manual/rdbms.md index 3a7525a..26feecf 100644 --- a/doc/user-manual/rdbms.md +++ b/doc/user-manual/rdbms.md @@ -108,7 +108,7 @@ You'll need to make several changes: This amounts to manually importing into the database the XML files that define the pipeline, parameters, and data types. Fortunately, there are ziggy commands you can use for all of these actions: - The command `ziggy import-parameters` allows you to read in the parameter library files. -- The command `ziggy import-types` allows you to read in the data file type definitions. +- The command `ziggy import-datastore-config` allows you to read in the data file type definitions and the definition of the datastore layout. - The command `ziggy import-pipelines` allows you to read in the pipeline definitions. All of the commands listed above will allow you to get help to know the exact syntax, order of arguments, etc. For more information on the ziggy program, take a look at [the article on running the cluster](running-pipeline.md). Most importantly: **Be sure to run the commands in the order shown above**, and specifically **be sure to run import-pipelines last!** diff --git a/doc/user-manual/redefine-pipeline.md b/doc/user-manual/redefine-pipeline.md index b81527f..1c4b5c4 100644 --- a/doc/user-manual/redefine-pipeline.md +++ b/doc/user-manual/redefine-pipeline.md @@ -12,7 +12,7 @@ The good news is that it's straightforward to update a pipeline definition in Zi To see this in action, open up the Pipelines panel, select the sample pipeline, and run the `View` command from the context menu. You'll see this: - + This shows the modules in the pipeline and their order of execution. @@ -30,7 +30,7 @@ $ ziggy update-pipelines sample-pipeline/config-extra/pd-sample.xml **Refresh the Pipelines Panel:** Press the `Refresh` button in the pipelines panel. Again, select the sample pipeline, and run the `View` command from the context menu. You'll now see this: - + As advertised, the averaging module has been removed from the end of the pipeline. diff --git a/doc/user-manual/rerun-task.md b/doc/user-manual/rerun-task.md index 181deff..8f55079 100644 --- a/doc/user-manual/rerun-task.md +++ b/doc/user-manual/rerun-task.md @@ -34,10 +34,6 @@ This tells Ziggy to try to pick up where it left off, in effect to force the pro Generally the case where you'd use this is the one we're in now: some subtasks ran, others didn't, the task made it to `WAITING_TO_STORE` and then halted. Selecting this option will cause Ziggy to advance to `STORING`, at which point it will store the results from the 3 successful subtasks and abandon efforts to get results from the failed one. -#### Resume Monitoring - -In somewhat unusual cases, it may happen that the monitoring subsystem will lose track of what's going on with one or more tasks. In this case, you may see signs that execution is progressing, but the console doesn't show any updates. In this case, the `Resume monitoring` option tells the monitoring subsystem to try to reconnect with the running task. - ### Restarting Multiple Tasks If you have a bunch of tasks for a given node, it may be that some will fail while others don't, and you want to restart all the failed tasks. There are two ways to do this. @@ -54,7 +50,7 @@ Why not? If the subtask had failed because of a real problem, we would be able to fix the problem and resubmit the task, or restart from the beginning. But what actually happened is that we set a module parameter that told `permuter` to throw an exception in subtask 0. -If we re-run the task, it will re-run with the same values for all parameters (except for the remote execution parameters, but we're not using those at all for this example). This means that the `throw exception subtask 0 parameter` will still be true, and subtask 0 will fail again. +If we re-run the task, it will re-run with the same values for all parameters. This means that the `throw exception subtask 0 parameter` will still be true, and subtask 0 will fail again. In real life, it's possible that you'll encounter a situation like this one, in which a task fails and the only way to get it to run successfully is to change the values of some module parameters. In that case, you won't be able to re-run because re-run doesn't let you change the parameters. In that case, you'll need to change the parameters and use the pipelines panel to start a new pipeline instance. In the more common cases (software bug that had to be fixed, failure due to some sort of hardware problem, etc.), re-running a task offers the possibility of getting failed subtasks to run to completion. For example, in this case we could simulate "fixing" the problem updating the code to ignore the `throw exception subtask 0 parameter`. diff --git a/doc/user-manual/running-pipeline.md b/doc/user-manual/running-pipeline.md index 386a740..f5e5f8a 100644 --- a/doc/user-manual/running-pipeline.md +++ b/doc/user-manual/running-pipeline.md @@ -30,6 +30,7 @@ dump-system-properties gov.nasa.ziggy.services.config.DumpSystemProperties execsql gov.nasa.ziggy.services.database.SqlRunner export-parameters gov.nasa.ziggy.pipeline.xml.ParameterLibraryImportExportCli export-pipelines gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli +generate-build-info gov.nasa.ziggy.util.BuildInfo generate-manifest gov.nasa.ziggy.data.management.Manifest hsqlgui org.hsqldb.util.DatabaseManagerSwing import-datastore-config gov.nasa.ziggy.data.management.DatastoreConfigurationImporter @@ -38,11 +39,14 @@ import-parameters gov.nasa.ziggy.pipeline.xml.ParameterLibraryImportExpor import-pipelines gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli metrics gov.nasa.ziggy.metrics.report.MetricsCli perf-report gov.nasa.ziggy.metrics.report.PerformanceReport -$ +update-pipelines gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli +$ ``` You can view more help with `ziggy --help` and even more help with `perldoc ziggy`. +Since there are a lot of commands, sub-commands, and options, we've created a bash completions file for the `ziggy` program so you can press the `TAB` key while entering the `ziggy` program to display the available commands and options. If you want to use it, run `. $ZIGGY_ROOT/etc/ziggy.bash-completion`. That's a dot at the front; it's the same mechanism that you would use to re-read your `.bashrc` file. + If you should happen to write some Java to manage your pipeline and want to use the `ziggy` program to run it, please refer to the article on [Creating Ziggy Nicknames](nicknames.md). ### Ziggy Cluster Commands @@ -104,7 +108,7 @@ That said: if your cluster initialization fails because of a problem in the XML, If the failure was in the import of the contents of the pipeline-defining XML files, there's an alternative to using `ziggy cluster init`. Specifically, you can use other ziggy commands that import the XML files without performing initialization. -If you look at the list of ziggy nicknames in the top screen shot, there are 3 that will be helpful here: `import-parameters`, `import-types`, and `import-pipelines`. These do what they say: import the parameter library, data type definition, and pipeline definition files, respectively. +If you look at the list of ziggy nicknames in the top screen shot, there are 3 that will be helpful here: `import-parameters`, `import-datastore-config`, and `import-pipelines`. These do what they say: import the parameter library, data type definition, and pipeline definition files, respectively. Important note: if you decide to manually import the XML files, **you must do so in the order shown above:** parameters, then data types, then the pipeline definitions. This is because some items can't import correctly unless other items that they depend upon have already been pulled in. diff --git a/doc/user-manual/user-manual.md b/doc/user-manual/user-manual.md index 343f3b8..598d46a 100644 --- a/doc/user-manual/user-manual.md +++ b/doc/user-manual/user-manual.md @@ -138,7 +138,9 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 19.6. [Advanced Unit of Work Configurations](advanced-uow.md) - + 19.7. [Software Version Tracking](version-tracking.md) + + 20. [Contact Us](contact-us.md) diff --git a/doc/user-manual/version-tracking.md b/doc/user-manual/version-tracking.md new file mode 100644 index 0000000..59f35ca --- /dev/null +++ b/doc/user-manual/version-tracking.md @@ -0,0 +1,36 @@ + + +[[Previous]](advanced-uow.md) +[[Up]](dusty-corners.md) +[[Next]](contact-us.md) + +## Software Version Tracking + +One of the key features of data accountability is knowledge of what version of your software (and our software!) was used to process any given task. + +When Ziggy starts a pipeline task, it automatically puts version information into the task's database entry, and the version is automatically updated any time the task is restarted. This is done by reading in properties from the `ziggy-build.properties` file in Ziggy's `build/etc` directory. Here's an example of what that file looks like: + +``` +# This file is automatically generated by Ziggy. +# Do not edit. +ziggy.version = Ziggy-0.6.0-20240726-323-g75138d913c +ziggy.version.branch = feature/ZIGGY-430-pipeline-version +ziggy.version.commit = 75138d913c +``` + +The `ziggy-build.properties` file, in turn, is constructed at the end of Ziggy's build. There's a Java program that runs a few Git commands, captures the output, and uses them to construct the build properties file. Whenever a pipeline task starts or restarts, it reads the file and stores the content of `ziggy-build.properties`, obtains the value of `ziggy.version`, and stores it in the database. + +Now, this is all well and good, but you are undoubtedly more interested in the version of the pipeline software that was used for a given task! After all, it's the pipeline software that contains the algorithms, and the pipeline results are going to be far more sensitive to changes in the algorithms than changes in Ziggy. Fortunately, Ziggy provides a method to perform a similar capture of Git information for a repository of pipeline files. + +To make this work, you need to first ensure that the `ziggy.pipeline.classpath` in the pipeline properties file includes the entry `${ziggy.home.dir}/libs/*` (for a gentle refresher on the properties files, take a look at [the article on configuring a pipeline](configuring-pipeline.md), and [the appendix article on properties files](properties.md)). Once you've done that, you can use the Ziggy command `$ZIGGY_HOME/bin/ziggy generate-build-info`, which will generate the file `pipeline-build.properties`. The resulting file will go in an `etc` subdirectory of whatever directory you set up as the pipeline's home directory (i.e., the directory that property `ziggy.pipeline.home.dir` points to). When Ziggy runs a pipeline task, it will automatically pick up the pipeline version from this file and store it in the database along with the Ziggy version. + +We recommend that you include this command that generates the pipeline build file in your build system. All build systems we're familiar with have an option for a build target / task that runs a shell command; you can use that capability to run the command above. + +Note that this tool will only work with Git repositories. If you're using something other than Git, and would like to track your software versions, [contact us](contact-us.md) and we'll put one together for you. + +Alternately, you can write a step into your own build system that generates a `pipeline-build.properties` file when you perform your build. As described above, the output of this build step must be put into the `etc` subdirectory of the directory specified by the `ziggy.pipeline.home.dir` property. The `pipeline-build.properties` file must contain a line that defines property `pipeline.version`; you can set this equal to any text string that makes sense, based on your version control system. You can also set the properties `pipeline.version.branch` and `pipeline.version.commit` if you have values for these that make sense, or you can leave them out. + +[[Previous]](advanced-uow.md) +[[Up]](dusty-corners.md) +[[Next]](contact-us.md) + diff --git a/etc/log4j2.xml b/etc/log4j2.xml index 6ed0e37..63f64d2 100644 --- a/etc/log4j2.xml +++ b/etc/log4j2.xml @@ -45,6 +45,9 @@ + + + @@ -52,10 +55,6 @@ - - - diff --git a/etc/ziggy.bash-completion b/etc/ziggy.bash-completion new file mode 100644 index 0000000..3757942 --- /dev/null +++ b/etc/ziggy.bash-completion @@ -0,0 +1,128 @@ +# ziggy completion -*- shell-script -*- +# +# To add Bash completion for the ziggy command, run ". $ZIGGY_ROOT/etc/ziggy.bash-completion". + +_ziggy() { + local cur prev commands options + cur="${COMP_WORDS[COMP_CWORD]}" + prev="${COMP_WORDS[COMP_CWORD-1]}" + + # Complete Ziggy nicknames and their commands or options. + case $prev in + ziggy) + nicknames=$(ziggy | tail -n +2 | awk '{print $1}') + COMPREPLY=($(compgen -W "${nicknames}" -- ${cur})) + return;; + + cluster) + commands="init start stop status console version -h --help" + COMPREPLY=($(compgen -W "${commands}" -- ${cur})) + return;; + + compute-node-master) + return;; + + console) + if [ ${COMP_CWORD} -eq 2 ]; then + commands="config display halt log restart start version" + COMPREPLY=($(compgen -W "${commands}" -- ${cur})) + fi + return;; + + dump-system-properties) + return;; + + execsql) + return;; + + export-parameters | export-pipelines | import-parameters | import-pipelines) + options="-dryrun -nodb" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + + generate-manifest) + return;; + + hsqlgui) + options="--help --driver --url --user --password --urlid --rcfile --dir --script --noexit" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + + import-datastore-config) + return;; + + import-events) + return;; + + metrics) + commands="available dump report" + COMPREPLY=($(compgen -W "${commands}" -- ${cur})) + return;; + + perf-report) + options="-force -id -nodes -taskdir" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + + update-pipelines) + options="-dryrun" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + esac + + # Complete sub-command options. + case "${COMP_WORDS[1]}" in + cluster) + case "${COMP_WORDS[2]}" in + init) + options="-f --force" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + start) + options="console --workerCount --workerHeapSize" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + esac;; + console) + case "${prev}" in + --configType) + options="data-model-registry instance pipeline pipeline-nodes" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + --displayType) + options="alerts errors full statistics statistics-detailed" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + --restartMode) + options="restart-from-beginning resume-current-step resubmit" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + esac + case "${COMP_WORDS[2]}" in + config) + options="--configType --instance --pipeline" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + display) + options="--displayType --instance --task" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + halt) + options="--instance --task" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + log) + options="--task --errors" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + restart) + options="--restartMode --instance --task" + COMPREPLY=($(compgen -W "${options}" -- ${cur})) + return;; + esac;; + esac +} + +complete -F _ziggy ziggy + + diff --git a/etc/ziggy.properties b/etc/ziggy.properties index da19242..2524d7d 100644 --- a/etc/ziggy.properties +++ b/etc/ziggy.properties @@ -18,6 +18,7 @@ ziggy.nickname.dump-system-properties = gov.nasa.ziggy.services.config.DumpSyste ziggy.nickname.execsql = gov.nasa.ziggy.services.database.SqlRunner||| ziggy.nickname.export-parameters = gov.nasa.ziggy.pipeline.xml.ParameterLibraryImportExportCli|||-export ziggy.nickname.export-pipelines = gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli|||-export +ziggy.nickname.generate-build-info = gov.nasa.ziggy.util.BuildInfo||| ziggy.nickname.generate-manifest = gov.nasa.ziggy.data.management.Manifest||| ziggy.nickname.hsqlgui = org.hsqldb.util.DatabaseManagerSwing||| ziggy.nickname.import-datastore-config = gov.nasa.ziggy.data.management.DatastoreConfigurationImporter||| diff --git a/gradle.properties b/gradle.properties index ab5b16e..ae43c4a 100644 --- a/gradle.properties +++ b/gradle.properties @@ -7,7 +7,7 @@ org.gradle.parallel = true // The version is updated when the first release candidate is created // while following Release Branches in Appendix C of the SMP, Git // Workflow. This property is used when publishing Ziggy. -version = 0.6.0 +version = 0.7.0 // The Maven group for the published Ziggy libraries. group = gov.nasa diff --git a/sample-pipeline/build-env.sh b/sample-pipeline/build-env.sh index 638c3d0..ebc639d 100755 --- a/sample-pipeline/build-env.sh +++ b/sample-pipeline/build-env.sh @@ -45,7 +45,7 @@ source $python_env/bin/activate pip3 install h5py Pillow numpy # Get the location of the environment's site packages directory -site_pkgs=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") +site_pkgs=$(python3 -c "from sysconfig import get_path; print(get_path('purelib'))") # Copy the pipeline major_tom package to the site-packages location. cp -r $ziggy_root/sample-pipeline/src/main/python/major_tom $site_pkgs @@ -54,4 +54,7 @@ cp -r $ziggy_root/sample-pipeline/src/main/python/major_tom $site_pkgs cp -r $ziggy_root/src/main/python/hdf5mi $site_pkgs cp -r $ziggy_root/src/main/python/zigutils $site_pkgs +# Generate version information. +$ZIGGY_HOME/bin/ziggy generate-build-info --home $sample_home + exit 0 diff --git a/sample-pipeline/src/main/sh/averaging.sh b/sample-pipeline/src/main/sh/averaging.sh index 8f58b59..24061bc 100755 --- a/sample-pipeline/src/main/sh/averaging.sh +++ b/sample-pipeline/src/main/sh/averaging.sh @@ -39,7 +39,7 @@ trap 'deactivate' EXIT source $SAMPLE_PIPELINE_PYTHON_ENV/bin/activate # Get the location of the environment's site packages directory -SITE_PKGS=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") +SITE_PKGS=$(python3 -c "from sysconfig import get_path; print(get_path('purelib'))") # Use the environment's Python to run the image-averaging Python script python3 $SITE_PKGS/major_tom/averaging.py diff --git a/sample-pipeline/src/main/sh/flipper.sh b/sample-pipeline/src/main/sh/flipper.sh index 64ed740..22799a1 100755 --- a/sample-pipeline/src/main/sh/flipper.sh +++ b/sample-pipeline/src/main/sh/flipper.sh @@ -39,7 +39,7 @@ trap 'deactivate' EXIT source $SAMPLE_PIPELINE_PYTHON_ENV/bin/activate # Get the location of the environment's site packages directory -SITE_PKGS=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") +SITE_PKGS=$(python3 -c "from sysconfig import get_path; print(get_path('purelib'))") # Use the environment's Python to run the flipper Python script python3 $SITE_PKGS/major_tom/flipper.py diff --git a/sample-pipeline/src/main/sh/permuter.sh b/sample-pipeline/src/main/sh/permuter.sh index 0d608d3..07f0154 100755 --- a/sample-pipeline/src/main/sh/permuter.sh +++ b/sample-pipeline/src/main/sh/permuter.sh @@ -39,7 +39,7 @@ trap 'deactivate' EXIT source $SAMPLE_PIPELINE_PYTHON_ENV/bin/activate # Get the location of the environment's site packages directory. -SITE_PKGS=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") +SITE_PKGS=$(python3 -c "from sysconfig import get_path; print(get_path('purelib'))") # Use the environment's Python to run the permuter Python script. python3 $SITE_PKGS/major_tom/permuter.py diff --git a/script-plugins/database-schemas.gradle b/script-plugins/database-schemas.gradle index ac02061..0261786 100644 --- a/script-plugins/database-schemas.gradle +++ b/script-plugins/database-schemas.gradle @@ -13,9 +13,9 @@ task generateHsqldbCreateScript(type: JavaExec, dependsOn: copyLibs) { mainClass.set("gov.nasa.ziggy.services.database.ZiggySchemaExport") classpath fileTree(dir: "$buildDir/libs", include: "*.jar") - jvmArgs "-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect", - "-Djava.library.path=$outsideDir/lib", - "-Dhibernate.connection.url=jdbc:hsqldb:mem:test" + jvmArgs "-Dhibernate.connection.url=jdbc:hsqldb:mem:test", + "-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect", + "-Djava.library.path=$outsideDir/lib" args "--create", "--output=$buildDir/schema/ddl.hsqldb-create.sql" logging.captureStandardOutput LogLevel.INFO @@ -23,6 +23,7 @@ task generateHsqldbCreateScript(type: JavaExec, dependsOn: copyLibs) { } test.dependsOn generateHsqldbCreateScript +integrationTest.dependsOn generateHsqldbCreateScript assemble.dependsOn generateHsqldbCreateScript task generateHsqldbDropScript(type: JavaExec, dependsOn: copyLibs) { @@ -32,9 +33,9 @@ task generateHsqldbDropScript(type: JavaExec, dependsOn: copyLibs) { mainClass.set("gov.nasa.ziggy.services.database.ZiggySchemaExport") classpath fileTree(dir: "$buildDir/libs", include: "*.jar") - jvmArgs "-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect", - "-Djava.library.path=$outsideDir/lib", - "-Dhibernate.connection.url=jdbc:hsqldb:mem:test" + jvmArgs "-Dhibernate.connection.url=jdbc:hsqldb:mem:test", + "-Dhibernate.dialect=org.hibernate.dialect.HSQLDialect", + "-Djava.library.path=$outsideDir/lib" args "--drop", "--output=$buildDir/schema/ddl.hsqldb-drop.sql" logging.captureStandardOutput LogLevel.INFO @@ -42,6 +43,7 @@ task generateHsqldbDropScript(type: JavaExec, dependsOn: copyLibs) { } test.dependsOn generateHsqldbDropScript +integrationTest.dependsOn generateHsqldbDropScript assemble.dependsOn generateHsqldbDropScript task generatePostgresqlCreateScript(type: JavaExec, dependsOn: copyLibs) { @@ -51,9 +53,9 @@ task generatePostgresqlCreateScript(type: JavaExec, dependsOn: copyLibs) { mainClass.set("gov.nasa.ziggy.services.database.ZiggySchemaExport") classpath fileTree(dir: "$buildDir/libs", include: "*.jar") - jvmArgs "-Dhibernate.dialect=org.hibernate.dialect.PostgreSQLDialect", - "-Djava.library.path=$outsideDir/lib", - "-Dhibernate.connection.url=jdbc:hsqldb:mem:test" + jvmArgs "-Dhibernate.connection.url=jdbc:hsqldb:mem:test", + "-Dhibernate.dialect=org.hibernate.dialect.PostgreSQLDialect", + "-Djava.library.path=$outsideDir/lib" args "--create", "--output=$buildDir/schema/ddl.postgresql-create.sql" logging.captureStandardOutput LogLevel.INFO @@ -69,9 +71,9 @@ task generatePostgresqlDropScript(type: JavaExec, dependsOn: copyLibs) { mainClass.set("gov.nasa.ziggy.services.database.ZiggySchemaExport") classpath fileTree(dir: "$buildDir/libs", include: "*.jar") - jvmArgs "-Dhibernate.dialect=org.hibernate.dialect.PostgreSQLDialect", - "-Djava.library.path=$outsideDir/lib", - "-Dhibernate.connection.url=jdbc:hsqldb:mem:test" + jvmArgs "-Dhibernate.connection.url=jdbc:hsqldb:mem:test", + "-Dhibernate.dialect=org.hibernate.dialect.PostgreSQLDialect", + "-Djava.library.path=$outsideDir/lib" args "--drop", "--output=$buildDir/schema/ddl.postgresql-drop.sql" logging.captureStandardOutput LogLevel.INFO diff --git a/script-plugins/misc.gradle b/script-plugins/misc.gradle index a3b3a73..a09b74f 100644 --- a/script-plugins/misc.gradle +++ b/script-plugins/misc.gradle @@ -1,9 +1,6 @@ // Show all dependencies. task allDeps(type: DependencyReportTask) {} -// Generate the Ziggy version information. -import gov.nasa.ziggy.buildutil.ZiggyVersionGenerator - def gitVersion = new ByteArrayOutputStream() exec { commandLine "git", "rev-parse", "HEAD" @@ -11,14 +8,16 @@ exec { } gitVersion = gitVersion.toString().trim() -task ziggyVersion(type: ZiggyVersionGenerator) { +task ziggyVersion(type: JavaExec) { inputs.property "ziggyVersion", gitVersion + outputs.file "$buildDir/etc/ziggy-build.properties" + classpath = sourceSets.main.runtimeClasspath + mainClass = "gov.nasa.ziggy.util.BuildInfo" + args = ["--prefix", "ziggy", "--home", "$buildDir"] } -processResources.dependsOn ziggyVersion -compileTestJava.dependsOn processResources -integrationTest.dependsOn processResources -sourcesJar.dependsOn processResources +integrationTest.dependsOn ziggyVersion +assemble.dependsOn ziggyVersion clean.doFirst() { File supervisorPidFile = new File("$buildDir/bin/supervisor.pid"); diff --git a/script-plugins/test.gradle b/script-plugins/test.gradle index 834029d..f8340e0 100644 --- a/script-plugins/test.gradle +++ b/script-plugins/test.gradle @@ -1,3 +1,73 @@ +test { + systemProperty "java.library.path", "$outsideDir/lib" + maxHeapSize = "1024m" + + testLogging { + events "failed", "skipped" + } + useJUnit { + // If a code coverage report that incudes the integration tests is desired, then comment + // out the IntegrationTestCategory line and uncomment the RunByNameTestCategory line. When + // the JaCoCo issue described below is resolved, then delete this comment. + // excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory' + excludeCategories 'gov.nasa.ziggy.IntegrationTestCategory' + } + + // Use "gradle -P traceTests test" to show test order. + if (project.hasProperty("traceTests")) { + afterTest { desc, result -> + logger.quiet "${desc.className}.${desc.name}: ${result.resultType}" + } + } +} + +// Execute tests marked with @Category(IntegrationTestCategory.class). +task integrationTest(type: Test) { + systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml" + systemProperty "ziggy.logfile", "$buildDir/build.log" + systemProperty "java.library.path", "$outsideDir/lib" + + testLogging { + events "failed", "skipped" + } + useJUnit { + includeCategories 'gov.nasa.ziggy.IntegrationTestCategory' + excludeCategories 'gov.nasa.ziggy.FlakyTestCategory' + excludeCategories 'gov.nasa.ziggy.RunByNameTestCategory' + } +} + +check.dependsOn integrationTest +cleanTest.dependsOn cleanIntegrationTest + +// Execute tests marked with @Category(FlakyTestCategory.class). +task flakyTest(type: Test) { + systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml" + systemProperty "ziggy.logfile", "$buildDir/build.log" + systemProperty "java.library.path", "$outsideDir/lib" + + useJUnit { + includeCategories 'gov.nasa.ziggy.FlakyTestCategory' + } +} + +check.dependsOn flakyTest +cleanTest.dependsOn cleanFlakyTest + +// Execute tests marked with @Category(RunByNameTestCategory.class). +// These tests are typically run explicitly with the --tests option +// since they don't play well with others. For example: +// gradle runByNameTests --tests *RmiInterProcessCommunicationTest +task runByNameTest(type: Test) { + systemProperty "log4j2.configurationFile", "$rootDir/etc/log4j2.xml" + systemProperty "ziggy.logfile", "$buildDir/build.log" + systemProperty "java.library.path", "$outsideDir/lib" + + useJUnit { + includeCategories 'gov.nasa.ziggy.RunByNameTestCategory' + } +} + /** * testprog is used to test external process control. testprog has the * advantage over something like /bin/true that it can run for a specified diff --git a/script-plugins/xml-schemas.gradle b/script-plugins/xml-schemas.gradle index 5189e74..c3d70d8 100644 --- a/script-plugins/xml-schemas.gradle +++ b/script-plugins/xml-schemas.gradle @@ -1,5 +1,5 @@ // Generate XML schemas. -task generateXmlSchemas(type: JavaExec, dependsOn: [copyLibs, processResources]) { +task generateXmlSchemas(type: JavaExec, dependsOn: copyLibs) { outputs.dir "$projectDir/build/schema/xml" mainClass.set("gov.nasa.ziggy.pipeline.xml.XmlSchemaExporter") diff --git a/src/main/java/gov/nasa/ziggy/collections/ListChunkIterator.java b/src/main/java/gov/nasa/ziggy/collections/ListChunkIterator.java deleted file mode 100644 index 3cbbb5c..0000000 --- a/src/main/java/gov/nasa/ziggy/collections/ListChunkIterator.java +++ /dev/null @@ -1,52 +0,0 @@ -package gov.nasa.ziggy.collections; - -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; - -/** - * Given a list, returns successive sub-lists which encompass all of the elements of the list. Each - * sub-list is limited to some maximum size. - * - * @author Sean McCauliff - */ -public class ListChunkIterator implements Iterator>, Iterable> { - private final Iterator source; - private final int chunkSize; - - public ListChunkIterator(Iterator source, int chunkSize) { - if (source == null) { - throw new NullPointerException("source"); - } - this.source = source; - this.chunkSize = chunkSize; - } - - public ListChunkIterator(Iterable source, int chunkSize) { - this(source.iterator(), chunkSize); - } - - @Override - public boolean hasNext() { - return source.hasNext(); - } - - @Override - public List next() { - List rv = new ArrayList<>(chunkSize); - for (int i = 0; i < chunkSize && source.hasNext(); i++) { - rv.add(source.next()); - } - return rv; - } - - @Override - public void remove() { - throw new UnsupportedOperationException("Operation not supported."); - } - - @Override - public Iterator> iterator() { - return this; - } -} diff --git a/src/main/java/gov/nasa/ziggy/collections/ZiggyArrayUtils.java b/src/main/java/gov/nasa/ziggy/collections/ZiggyArrayUtils.java index 419eeed..4c44f54 100644 --- a/src/main/java/gov/nasa/ziggy/collections/ZiggyArrayUtils.java +++ b/src/main/java/gov/nasa/ziggy/collections/ZiggyArrayUtils.java @@ -145,6 +145,13 @@ public static long[] getArraySize(Object dataObject) { String componentType = loopObject.getClass().getComponentType().getName(); if (componentType.startsWith("[")) { Object[] arrayObject = (Object[]) loopObject; + + // Handle a zero-length array. + if (arrayObject == null || arrayObject.length == 0) { + arrayDimensionList.add(0L); + isArray = false; + continue; + } loopObject = arrayObject[0]; isArray = true; } else { diff --git a/src/main/java/gov/nasa/ziggy/crud/AbstractCrud.java b/src/main/java/gov/nasa/ziggy/crud/AbstractCrud.java index 84247b4..6cc28b3 100644 --- a/src/main/java/gov/nasa/ziggy/crud/AbstractCrud.java +++ b/src/main/java/gov/nasa/ziggy/crud/AbstractCrud.java @@ -1,13 +1,20 @@ package gov.nasa.ziggy.crud; +import java.util.ArrayList; import java.util.Collection; +import java.util.Collections; import java.util.List; +import java.util.function.Function; import org.hibernate.Session; import org.hibernate.query.Query; import org.hibernate.query.criteria.HibernateCriteriaBuilder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.collections.ListChunkIterator; +import com.google.common.collect.Lists; + +import gov.nasa.ziggy.services.database.DatabaseController; import gov.nasa.ziggy.services.database.DatabaseService; import jakarta.persistence.LockModeType; import jakarta.persistence.criteria.CriteriaBuilder; @@ -26,14 +33,7 @@ */ public abstract class AbstractCrud implements AbstractCrudInterface { - /** - * This is the maximum number of dynamically-created expressions sent to the database. This - * limit is 1000 in Oracle. A setting of 950 leaves plenty of room for other expressions in the - * query. - * - * @see ListChunkIterator - */ - public static final int MAX_EXPRESSIONS = 950; + private static final Logger log = LoggerFactory.getLogger(AbstractCrud.class); private DatabaseService databaseService; @@ -186,6 +186,48 @@ public List list(ZiggyQuery query) { .getResultList(); } + /** + * Performs a query in which the query must be broken into multiple discrete queries due to + * database query language limitations, the results of which are then combined and returned. + *

+ * For example: + * + * 

+     * chunkedQuery(pipelineTaskIds,
+     *     chunk -> list(createZiggyQuery(PipelineTask.class).column(PipelineTask_.id)
+     *         .ascendingOrder()
+     *         .in(chunk)));
+     *
+ * + * The variable constraintsCollection is the collection of objects of class T that constrain the + * query and queryWithRestraints is a method that returns a query that applies the constraints. + * + * @param class of the objects in the list that constrain the query + * @param class of the objects in the list returned by the query + * @param source the list of elements of type T to use in the query + * @param query returns a list of results of type R based upon the collection of type T + * @return list of type R + */ + protected List chunkedQuery(List source, Function, List> query) { + if (source.isEmpty()) { + return Collections.emptyList(); + } + int maxExpressions = maxExpressions(); + List results = new ArrayList<>(maxExpressions * 2); + for (List chunk : Lists.partition(source, maxExpressions)) { + log.info("Created chunk of size {}", chunk.size()); + results.addAll(query.apply(chunk)); + } + return results; + } + + /** + * Maximum expressions allowed in a query. + */ + int maxExpressions() { + return DatabaseController.newInstance().maxExpressions(); + } + /** * Flush any changes to persistent objects to the underlying database. */ diff --git a/src/main/java/gov/nasa/ziggy/crud/ProtectedEntityInterceptor.java b/src/main/java/gov/nasa/ziggy/crud/ProtectedEntityInterceptor.java index d8c3c0e..9bd24c5 100644 --- a/src/main/java/gov/nasa/ziggy/crud/ProtectedEntityInterceptor.java +++ b/src/main/java/gov/nasa/ziggy/crud/ProtectedEntityInterceptor.java @@ -22,7 +22,7 @@ public class ProtectedEntityInterceptor implements Interceptor { private static final List allowedPrefixes = new CopyOnWriteArrayList<>(); public static void addAllowedPrefix(String prefix) { - log.info("Adding allowed prefix for flushed classes: " + prefix); + log.info("Adding allowed prefix {} for flushed classes", prefix); allowedPrefixes.add(prefix); } diff --git a/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java b/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java index fefec3c..249ae80 100644 --- a/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java +++ b/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java @@ -7,11 +7,9 @@ import java.util.List; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.hibernate.query.criteria.HibernateCriteriaBuilder; -import com.google.common.collect.Lists; - import gov.nasa.ziggy.module.PipelineException; import jakarta.persistence.criteria.AbstractQuery; import jakarta.persistence.criteria.CriteriaBuilder; @@ -79,9 +77,6 @@ public class ZiggyQuery { // AbstractQuery allows this to be either CriteriaQuery or Subquery, as needed. private AbstractQuery jpaQuery; - /** For testing only. */ - private List> queryChunks = new ArrayList<>(); - /** Constructor for {@link CriteriaQuery} instances. */ ZiggyQuery(Class databaseClass, Class returnClass, AbstractCrud crud) { builder = crud.createCriteriaBuilder(); @@ -244,27 +239,6 @@ public ZiggyQuery in(Collection values) { return this; } - /** - * Performs the action of the {@link #in(Collection)} method, but performs the query in chunks. - * This allows queries in which the collection of values is too large for a single query. The - * number of values in a chunk is set by the {@link AbstractCrud#MAX_EXPRESSIONS} constant. - */ - @SuppressWarnings("unchecked") - public ZiggyQuery chunkedIn(Collection values) { - checkState(hasScalarAttribute(), "a column has not been defined"); - List valuesList = new ArrayList<>(values); - Predicate criterion = builder.disjunction(); - for (List valuesSubset : Lists.partition(valuesList, maxExpressions())) { - queryChunks.add((List) valuesSubset); - criterion = builder.or(criterion, - attribute != null - ? builder.in((Path) root.get(attribute), valuesSubset) - : builder.in(root.get(columnName), valuesSubset)); - } - predicates.add(criterion); - return this; - } - public CriteriaBuilder.In in(Expression expression, Collection values) { return builder.in(expression, values); } @@ -579,18 +553,4 @@ public CriteriaQuery getCriteriaQuery() { } return (CriteriaQuery) jpaQuery; } - - /** - * Maximum expressions allowed in each chunk of {@link #chunkedIn(Collection)}. Broken out into - * a package-private method so that tests can reduce this value to something small enough to - * exercise in test. - */ - int maxExpressions() { - return AbstractCrud.MAX_EXPRESSIONS; - } - - /** For testing only. */ - List> queryChunks() { - return queryChunks; - } } diff --git a/src/main/java/gov/nasa/ziggy/data/accounting/DetailedPipelineTaskRenderer.java b/src/main/java/gov/nasa/ziggy/data/accounting/DetailedPipelineTaskRenderer.java deleted file mode 100644 index a982a5e..0000000 --- a/src/main/java/gov/nasa/ziggy/data/accounting/DetailedPipelineTaskRenderer.java +++ /dev/null @@ -1,20 +0,0 @@ -package gov.nasa.ziggy.data.accounting; - -import gov.nasa.ziggy.pipeline.definition.PipelineTask; - -/** - * Writes out a task using PiplineTask.prettyPrint() - * - * @author Sean McCauliff - */ -public class DetailedPipelineTaskRenderer implements PipelineTaskRenderer { - @Override - public String renderTask(PipelineTask task) { - return task.prettyPrint(); - } - - @Override - public String renderDefaultTask() { - return "Data Receipt"; - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/accounting/SimpleTaskRenderer.java b/src/main/java/gov/nasa/ziggy/data/accounting/SimpleTaskRenderer.java index 5950e77..f059544 100644 --- a/src/main/java/gov/nasa/ziggy/data/accounting/SimpleTaskRenderer.java +++ b/src/main/java/gov/nasa/ziggy/data/accounting/SimpleTaskRenderer.java @@ -10,7 +10,7 @@ public class SimpleTaskRenderer implements PipelineTaskRenderer { @Override public String renderTask(PipelineTask task) { - return task.uowTaskInstance().briefState(); + return task.getUnitOfWork().briefState(); } @Override diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java index f5c8b3a..717e96a 100644 --- a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java @@ -51,7 +51,7 @@ import org.apache.commons.cli.Option; import org.apache.commons.cli.Options; import org.apache.commons.cli.ParseException; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -533,8 +533,9 @@ private void checkDataFileTypeDefinitions() { List databaseDataFileTypeNames = datastoreOperations().dataFileTypeNames(); for (DataFileType typeXb : importedDataFileTypes) { if (databaseDataFileTypeNames.contains(typeXb.getName())) { - log.warn("Not importing data file type definition \"{}\"" - + " due to presence of existing type with same name", typeXb.getName()); + log.warn( + "Not importing data file type definition {} due to presence of existing type with same name", + typeXb.getName()); dataFileTypesNotImported.add(typeXb); continue; } @@ -562,14 +563,13 @@ private void checkModelTypeDefinitions() { try { modelTypeXb.validate(); } catch (Exception e) { - log.warn("Unable to validate model type definition " + modelTypeXb.getType(), e); + log.warn("Unable to validate model type definition {}", modelTypeXb.getType(), e); modelTypesNotImported.add(modelTypeXb); continue; } if (databaseModelTypes.contains(modelTypeXb.getType())) { log.warn( - "Not importing model type definition \"{}\"" - + " due to presence of existing type with same name", + "Not importing model type definition {} due to presence of existing type with same name", modelTypeXb.getType()); modelTypesNotImported.add(modelTypeXb); continue; diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java index 50933d4..38eb6f0 100644 --- a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java @@ -39,10 +39,10 @@ import gov.nasa.ziggy.pipeline.definition.ModelType; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; @@ -128,7 +128,7 @@ public Set subtaskDefinitions( List allFilesAllSubtasksDataFileTypes = new ArrayList<>(dataFileTypes); allFilesAllSubtasksDataFileTypes.removeAll(filePerSubtaskDataFileTypes); - UnitOfWork uow = pipelineTask.uowTaskInstance(); + UnitOfWork uow = pipelineTask.getUnitOfWork(); // Generate sets of DataFilesForDataFileType instances. These provide the necessary // information for mapping files in the datastore into the files needed by each @@ -248,13 +248,12 @@ private void filterOutDataFilesAlreadyProcessed( .collect(Collectors.toSet()); // Find the consumers that correspond to the definition node of the current task. - List consumersWithMatchingPipelineNode = pipelineTaskOperations() - .taskIdsForPipelineDefinitionNode(pipelineTask); + List consumersWithMatchingPipelineNode = pipelineTaskOperations() + .tasksForPipelineDefinitionNode(pipelineTask); // Obtain the Set of datastore files that are in the relativizedFilePaths collection // and which have a consumer that matches the pipeline definition node of the - // current - // pipeline task. + // current pipeline task. Set namesOfFilesAlreadyProcessed = datastoreProducerConsumerOperations() .filesConsumedByTasks(consumersWithMatchingPipelineNode, relativizedFilePaths); diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java index 79bc6b3..bf15e32 100644 --- a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java @@ -18,7 +18,7 @@ import java.util.regex.Pattern; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java b/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java index ce4593b..a5041f6 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java +++ b/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java @@ -22,8 +22,8 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.xml.HasXmlSchemaFilename; import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -141,8 +141,8 @@ public static boolean validAcknowledgementExists(Path workingDir, Manifest manif if (!Files.exists(acknowledgementPath)) { return false; } - ValidatingXmlManager xmlManager; - xmlManager = new ValidatingXmlManager<>(Acknowledgement.class); + ValidatingXmlManager xmlManager = new ValidatingXmlManager<>( + Acknowledgement.class); Acknowledgement ack = xmlManager.unmarshal(acknowledgementPath.toFile()); return ack.transferStatus.equals(DataReceiptStatus.VALID); } @@ -171,12 +171,12 @@ public List namesOfValidFiles() { * * @param manifest {@link Manifest} file to be acknowledged. * @param dir Location of the files referenced in the manifest. - * @param taskId ID of the {@link PipelineTask} that is performing the manifest validation. + * @param pipelineTask {@link PipelineTask} that is performing the manifest validation. * @return {@link Acknowledgement} that includes validation status of all files referenced in * the manifest. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static Acknowledgement of(Manifest manifest, Path dir, long taskId) { + public static Acknowledgement of(Manifest manifest, Path dir, PipelineTask pipelineTask) { Acknowledgement acknowledgement = new Acknowledgement(manifest.getChecksumType()); acknowledgement.setDatasetId(manifest.getDatasetId()); @@ -211,8 +211,8 @@ public static Acknowledgement of(Manifest manifest, Path dir, long taskId) { // failures. if (validationFailures == 0) { AlertService.getInstance() - .generateAndBroadcastAlert("Data Receipt", taskId, Severity.WARNING, - "File validation errors encountered"); + .generateAndBroadcastAlert("Data Receipt", pipelineTask, + Severity.WARNING, "File validation errors encountered"); } validationFailures++; @@ -221,7 +221,7 @@ public static Acknowledgement of(Manifest manifest, Path dir, long taskId) { // which files failed prior to termination. if (validationFailures >= maxValidationFailures) { AlertService.getInstance() - .generateAndBroadcastAlert("Data Receipt", taskId, Severity.ERROR, + .generateAndBroadcastAlert("Data Receipt", pipelineTask, Severity.ERROR, "Exceeded " + maxValidationFailures + ", terminating"); break; } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptOperations.java b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptOperations.java index 687f77e..d9c702e 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptOperations.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptOperations.java @@ -39,7 +39,7 @@ public List dataReceiptInstances() { for (PipelineInstance pipelineInstance : pipelineInstances) { DataReceiptInstance dataReceiptInstance = new DataReceiptInstance(); dataReceiptInstance.setInstanceId(pipelineInstance.getId()); - dataReceiptInstance.setDate(pipelineInstance.getStartProcessingTime()); + dataReceiptInstance.setDate(pipelineInstance.getCreated()); dataReceiptInstance.setFailedImportCount( failedImportCrud().retrieveCountForInstance(pipelineInstance.getId())); diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java index e42b09d..67e1357 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java @@ -27,8 +27,8 @@ import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; @@ -86,7 +86,7 @@ public DataReceiptPipelineModule(PipelineTask pipelineTask, RunMode runMode) { // Get the top-level DR directory and the datastore root directory ImmutableConfiguration config = ZiggyConfiguration.getInstance(); dataReceiptDir = config.getString(PropertyName.DATA_RECEIPT_DIR.property()); - UnitOfWork uow = pipelineTask.uowTaskInstance(); + UnitOfWork uow = pipelineTask.getUnitOfWork(); dataReceiptTopLevelPath = Paths.get(dataReceiptDir).toAbsolutePath(); dataImportPathForTask = dataImportPathForTask(uow); } @@ -111,9 +111,9 @@ public boolean processTask() { } if (!containsNonHiddenFiles) { - log.warn("Directory " + dataImportPathForTask.toString() - + " contains no files, skipping DR"); - alertService().generateAndBroadcastAlert("DR", pipelineTask.getId(), Severity.WARNING, + log.warn("Directory {} contains no files, skipping DR", + dataImportPathForTask.toString()); + alertService().generateAndBroadcastAlert("DR", pipelineTask, Severity.WARNING, "Directory " + dataImportPathForTask.toString() + " contains no files"); return true; } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java index b2a7d1e..83ae855 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java @@ -13,7 +13,7 @@ import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -24,8 +24,8 @@ import gov.nasa.ziggy.models.ModelOperations; import gov.nasa.ziggy.pipeline.definition.ModelType; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -132,7 +132,7 @@ private boolean checkDatasetId() { /** Generates acknowledgement and returns true if transfer status is VALID. */ private boolean acknowledgeManifest() { - acknowledgement = Acknowledgement.of(manifest, dataImportPath, pipelineTask.getId()); + acknowledgement = Acknowledgement.of(manifest, dataImportPath, pipelineTask); // Write the acknowledgement to the directory. acknowledgement.write(dataImportPath); @@ -175,7 +175,8 @@ private List filesNotInManifest() { // be the set of all names in the manifest) List namesOfValidFiles = acknowledgement.namesOfValidFiles(); - Map regularFilesInDirTree = ZiggyFileUtils.regularFilesInDirTree(dataImportPath); + Map regularFilesInDirTree = ZiggyFileUtils + .regularFilesInDirTree(dataImportPath); List filenamesInDirTree = regularFilesInDirTree.keySet() .stream() .map(Path::toString) @@ -283,7 +284,7 @@ private void importDataFiles() { move(file, destinationFile); successfulImports.add(datastoreRoot.relativize(destinationFile)); } catch (IOException e) { - log.error("Failed to import file " + file.toString(), e); + log.error("Failed to import file {}", file.toString(), e); exceptionCount++; failedImports.add(datastoreRoot.relativize(destinationFile)); } @@ -333,8 +334,8 @@ private void importModelFiles() { failedImports.addAll(importer.getFailedImports()); log.warn("{} out of {} model files failed to import", importer.getFailedImports(), modelFilesForImport.size()); - alertService().generateAndBroadcastAlert("Data Receipt (DR)", pipelineTask.getId(), - AlertService.Severity.WARNING, "Failed to import " + importer.getFailedImports() + alertService().generateAndBroadcastAlert("Data Receipt (DR)", pipelineTask, + Severity.WARNING, "Failed to import " + importer.getFailedImports() + " model files (out of " + modelFilesForImport.size() + ")"); } } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java index b363649..6674e16 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java @@ -33,6 +33,9 @@ * {@link ModelRegistry} of the current versions of all models that is provided to a * {@link PipelineInstance} when the instance is created, and which can be exposed by the instance * report. + *

+ * A non-producing consumer is a consumer that failed to produce results from processing. It is + * indicated by the negative of the task ID. * * @author PT */ @@ -61,14 +64,22 @@ public class DatastoreProducerConsumer { public DatastoreProducerConsumer() { } - public DatastoreProducerConsumer(long producerId, String filename) { + public DatastoreProducerConsumer(PipelineTask producerPipelineTask, Path datastoreFile) { + this(producerPipelineTask, datastoreFile.toString()); + } + + public DatastoreProducerConsumer(PipelineTask producerPipelineTask, String filename) { + this(toProducerId(producerPipelineTask), filename); + } + + private DatastoreProducerConsumer(long producerId, String filename) { checkNotNull(filename, "filename"); this.filename = filename; this.producerId = producerId; } - public DatastoreProducerConsumer(PipelineTask pipelineTask, Path datastoreFile) { - this(pipelineTask.getId(), datastoreFile.toString()); + private static long toProducerId(PipelineTask producerPipelineTask) { + return producerPipelineTask != null ? producerPipelineTask.getId() : 0; } public void setFilename(String filename) { @@ -83,8 +94,8 @@ public long getProducer() { return producerId; } - public void setProducer(long producer) { - producerId = producer; + public void setProducer(PipelineTask producer) { + producerId = toProducerId(producer); } public Set getConsumers() { @@ -97,11 +108,19 @@ public Set getAllConsumers() { return consumers.stream().map(Math::abs).collect(Collectors.toSet()); } - public void setConsumers(Set consumers) { - this.consumers.addAll(consumers); + public void addConsumer(PipelineTask consumingPipelineTask) { + addConsumer(toProducerId(consumingPipelineTask)); + } + + /** + * Adds the given consumer to this object. A non-producing consumer is a consumer that failed to + * produce results from processing. + */ + public void addNonProducingConsumer(PipelineTask consumingPipelineTask) { + addConsumer(-toProducerId(consumingPipelineTask)); } - public void addConsumer(long consumer) { + private void addConsumer(long consumer) { consumers.add(consumer); } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java index ad43e06..97aef01 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java @@ -10,7 +10,7 @@ import java.util.function.Function; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import gov.nasa.ziggy.crud.AbstractCrud; import gov.nasa.ziggy.crud.ZiggyQuery; @@ -61,7 +61,7 @@ public void createOrUpdateProducer(PipelineTask pipelineTask, Collection d List datastoreProducerConsumers = retrieveOrCreate(pipelineTask, datastoreNames(datastoreFiles)); for (DatastoreProducerConsumer datastoreProducerConsumer : datastoreProducerConsumers) { - datastoreProducerConsumer.setProducer(pipelineTask.getId()); + datastoreProducerConsumer.setProducer(pipelineTask); merge(datastoreProducerConsumer); } } @@ -72,8 +72,8 @@ public List retrieveByFilename(Set datastoreFil } /** Retrieves the set of names of datastore files consumed by a specified pipeline task. */ - public Set retrieveFilesConsumedByTask(long taskId) { - return retrieveFilesConsumedByTasks(Set.of(taskId), null); + public Set retrieveFilesConsumedByTask(PipelineTask pipelineTask) { + return retrieveFilesConsumedByTasks(Set.of(pipelineTask), null); } /** @@ -82,16 +82,27 @@ public Set retrieveFilesConsumedByTask(long taskId) { * filenames collection will be included in the return; otherwise, all filenames that have a * consumer from the collection of consumer IDs will be included. */ - public Set retrieveFilesConsumedByTasks(Collection consumerIds, + public Set retrieveFilesConsumedByTasks(Collection consumers, Collection filenames) { + + if (CollectionUtils.isEmpty(filenames)) { + return new HashSet<>(list(filesConsumedByTasksQuery(consumers))); + } + return new HashSet<>(chunkedQuery(new ArrayList<>(filenames), + chunk -> list( + filesConsumedByTasksQuery(consumers).column(DatastoreProducerConsumer_.filename) + .in(chunk)))); + } + + private ZiggyQuery filesConsumedByTasksQuery( + Collection consumers) { + ZiggyQuery query = createZiggyQuery( DatastoreProducerConsumer.class, String.class); query.select(DatastoreProducerConsumer_.filename).distinct(true); - if (!CollectionUtils.isEmpty(filenames)) { - query.column(DatastoreProducerConsumer_.filename).chunkedIn(filenames); - } - addConsumerIdPredicates(query, consumerIds); - return new HashSet<>(list(query)); + addConsumerIdPredicates(query, + consumers.stream().map(PipelineTask::getId).collect(Collectors.toList())); + return query; } /** @@ -139,18 +150,21 @@ public List retrieveConsumedFiles(Set producedFiles) { Set producedFileDatastoreNames = datastoreNames(producedFiles); // Find the unique set of pipeline tasks that produced the output files. - ZiggyQuery producerQuery = createZiggyQuery( - DatastoreProducerConsumer.class, Long.class); - producerQuery.column(DatastoreProducerConsumer_.filename) - .chunkedIn(producedFileDatastoreNames); - producerQuery.column(DatastoreProducerConsumer_.producerId).select().distinct(true); + List producerIds = chunkedQuery(new ArrayList<>(producedFileDatastoreNames), + chunk -> list(createZiggyQuery(DatastoreProducerConsumer.class, Long.class) + .column(DatastoreProducerConsumer_.filename) + .in(chunk) + .column(DatastoreProducerConsumer_.producerId) + .select() + .distinct(true))); // Find and return the files that were consumed by the tasks that produced the // outputs. ZiggyQuery query = createZiggyQuery( - DatastoreProducerConsumer.class, String.class); - query.column(DatastoreProducerConsumer_.filename).select(); - addConsumerIdPredicates(query, list(producerQuery)); + DatastoreProducerConsumer.class, String.class) + .column(DatastoreProducerConsumer_.filename) + .select(); + addConsumerIdPredicates(query, producerIds); return list(query); } @@ -160,13 +174,12 @@ public void addConsumer(PipelineTask pipelineTask, Set datastoreNames) { return; } List dpcs = retrieveOrCreate(null, datastoreNames); - dpcs.stream().forEach(s -> addConsumer(s, pipelineTask.getId())); + dpcs.stream().forEach(s -> addConsumer(s, pipelineTask)); } /** * Adds a non-producing consumer to each of a set of datastore files. A non-producing consumer - * is a consumer that failed to produce results from processing. It is indicated by the negative - * of the task ID. + * is a consumer that failed to produce results from processing. */ public void addNonProducingConsumer(PipelineTask pipelineTask, Set datastoreNames) { if (datastoreNames == null || datastoreNames.isEmpty()) { @@ -174,16 +187,25 @@ public void addNonProducingConsumer(PipelineTask pipelineTask, Set datas } List datastoreProducerConsumers = retrieveOrCreate(null, datastoreNames); - datastoreProducerConsumers.stream().forEach(dpc -> addConsumer(dpc, -pipelineTask.getId())); + datastoreProducerConsumers.stream() + .forEach(dpc -> addNonProducingConsumer(dpc, pipelineTask)); } /** - * Adds a consumer ID and updates or creates a {@link DatastoreProducerConsumer} instance. - * Implemented as a private method so that a stream forEach operation can apply it. + * Adds a consumer to the given {@link DatastoreProducerConsumer} instance. */ private void addConsumer(DatastoreProducerConsumer datastoreProducerConsumer, - long pipelineTaskId) { - datastoreProducerConsumer.addConsumer(pipelineTaskId); + PipelineTask pipelineTask) { + datastoreProducerConsumer.addConsumer(pipelineTask); + merge(datastoreProducerConsumer); + } + + /** + * Adds a consumer to the given {@link DatastoreProducerConsumer} instance. + */ + private void addNonProducingConsumer(DatastoreProducerConsumer datastoreProducerConsumer, + PipelineTask pipelineTask) { + datastoreProducerConsumer.addNonProducingConsumer(pipelineTask); merge(datastoreProducerConsumer); } @@ -199,29 +221,29 @@ private Set datastoreNames(Collection datastoreFiles) { * @param pipelineTask Producer task for constructed entries, if null a value of zero will be * used for the producer ID of constructed entries * @param filenames Names of datastore files to be located in the database table of - * {@link DatastoreProducerConsumer} instances. Must be mutable. + * {@link DatastoreProducerConsumer} instances. * @return A {@link List} of {@link DatastoreProducerConsumer} instances, with the database - * versions for files that have database entries and new instances for those that do not. + * versions for files that have database entries and new instances for those that do not */ protected List retrieveOrCreate(PipelineTask pipelineTask, Set filenames) { - Set allFilenames = new HashSet<>(filenames); // Start by finding all the files that already have entries. - ZiggyQuery query = createZiggyQuery( - DatastoreProducerConsumer.class); - query.column(DatastoreProducerConsumer_.filename).chunkedIn(allFilenames); - List datastoreProducerConsumers = list(query); + Set allFilenames = new HashSet<>(filenames); + List datastoreProducerConsumers = chunkedQuery( + new ArrayList<>(allFilenames), + chunk -> list(createZiggyQuery(DatastoreProducerConsumer.class) + .column(DatastoreProducerConsumer_.filename) + .in(chunk))); List locatedFilenames = datastoreProducerConsumers.stream() .map(DatastoreProducerConsumer::getFilename) .collect(Collectors.toList()); - // For all the filenames that lack entries, construct DatastoreProducerConsumer instances - long producerId = pipelineTask != null ? pipelineTask.getId() : 0; + // For all the filenames that lack entries, construct DatastoreProducerConsumer instances. allFilenames.removeAll(locatedFilenames); for (String filename : allFilenames) { - DatastoreProducerConsumer instance = new DatastoreProducerConsumer(producerId, + DatastoreProducerConsumer instance = new DatastoreProducerConsumer(pipelineTask, filename); persist(instance); datastoreProducerConsumers.add(instance); @@ -276,15 +298,12 @@ public Set newFiles(Collection datastoreFiles) { Map datastoreFileByName = datastoreFiles.stream() .collect(Collectors.toMap(Path::toString, Function.identity())); Set datastoreNames = new HashSet<>(datastoreFileByName.keySet()); - if (datastoreNames.iterator().hasNext()) { - } - // There doesn't seem to be a better way to do this. - datastoreNames - .removeAll(list(createZiggyQuery(DatastoreProducerConsumer.class, String.class) + datastoreNames.removeAll(chunkedQuery(new ArrayList<>(datastoreNames), + chunk -> list(createZiggyQuery(DatastoreProducerConsumer.class, String.class) .column(DatastoreProducerConsumer_.filename) - .chunkedIn(datastoreNames) - .select())); + .in(chunk) + .select()))); return datastoreNames.stream().map(datastoreFileByName::get).collect(Collectors.toSet()); } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperations.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperations.java index 531f945..76d79d9 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperations.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperations.java @@ -21,10 +21,10 @@ public class DatastoreProducerConsumerOperations extends DatabaseOperations { private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud = new DatastoreProducerConsumerCrud(); private PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); - public Set filesConsumedByTasks(Collection consumerIds, + public Set filesConsumedByTasks(Collection consumers, Collection filenames) { return performTransaction(() -> datastoreProducerConsumerCrud() - .retrieveFilesConsumedByTasks(consumerIds, filenames)); + .retrieveFilesConsumedByTasks(consumers, filenames)); } public void createOrUpdateProducer(PipelineTask pipelineTask, Collection files) { diff --git a/src/main/java/gov/nasa/ziggy/metrics/Metric.java b/src/main/java/gov/nasa/ziggy/metrics/Metric.java index 2529de8..6d130d3 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/Metric.java +++ b/src/main/java/gov/nasa/ziggy/metrics/Metric.java @@ -104,19 +104,20 @@ public static void clear() { */ public static void log() { long now = System.currentTimeMillis(); - metricsLogger.info("SNAPSHOT-START@" + now); + metricsLogger.info("SNAPSHOT-START@{}", now); Iterator it = Metric.metricsIterator(); logWithIterator(it); - metricsLogger.info("SNAPSHOT-END@" + now); + metricsLogger.info("SNAPSHOT-END@{}", now); } /** * Dump all metrics to stdout */ public static void dump() { - Metric.dump(new PrintWriter(new OutputStreamWriter(System.out, ZiggyFileUtils.ZIGGY_CHARSET))); + Metric.dump( + new PrintWriter(new OutputStreamWriter(System.out, ZiggyFileUtils.ZIGGY_CHARSET))); } /** @@ -197,13 +198,13 @@ public static void merge(String path) { for (String metricName : metricsToMerge.keySet()) { Metric metricToMerge = metricsToMerge.get(metricName); - log.debug("merge: metricToMerge=" + metricToMerge); + log.debug("metricToMerge={}", metricToMerge); Metric existingGlobalMetric = globalMetrics.get(metricName); if (existingGlobalMetric != null) { - log.debug("merge: existingGlobalMetric(BEFORE)=" + existingGlobalMetric); + log.debug("existingGlobalMetric(BEFORE)={}", existingGlobalMetric); existingGlobalMetric.merge(metricToMerge); - log.debug("merge: existingGlobalMetric(AFTER)=" + existingGlobalMetric); + log.debug("existingGlobalMetric(AFTER)={}", existingGlobalMetric); } else { log.debug("No existingGlobalMetric exists, adding"); globalMetrics.put(metricName, metricToMerge.makeCopy()); @@ -212,9 +213,9 @@ public static void merge(String path) { if (Metric.threadMetricsEnabled()) { Metric existingThreadMetric = Metric.getThreadMetric(metricName); if (existingThreadMetric != null) { - log.debug("merge: existingThreadMetric(BEFORE)=" + existingThreadMetric); + log.debug("existingThreadMetric(BEFORE)={}", existingThreadMetric); existingThreadMetric.merge(metricToMerge); - log.debug("merge: existingThreadMetric(AFTER)=" + existingThreadMetric); + log.debug("existingThreadMetric(AFTER)={}", existingThreadMetric); } else { log.debug("No existingThreadMetric exists, adding"); Metric.addNewThreadMetric(metricToMerge.makeCopy()); @@ -361,12 +362,12 @@ protected static Metric addNewThreadMetric(Metric metric) { */ protected static void log(String prefix) { long now = System.currentTimeMillis(); - metricsLogger.info("SNAPSHOT-START@" + now); + metricsLogger.info("SNAPSHOT-START@{}", now); Iterator it = Metric.metricsIterator(prefix); logWithIterator(it); - metricsLogger.info("SNAPSHOT-END@" + now); + metricsLogger.info("SNAPSHOT-END@{}", now); } /** @@ -376,14 +377,14 @@ protected static void log(String prefix) { */ protected static void log(List prefixes) { long now = System.currentTimeMillis(); - metricsLogger.info("SNAPSHOT-START@" + now); + metricsLogger.info("SNAPSHOT-START@{}", now); for (String prefix : prefixes) { Iterator it = Metric.metricsIterator(prefix); logWithIterator(it); } - metricsLogger.info("SNAPSHOT-END@" + now); + metricsLogger.info("SNAPSHOT-END@{}", now); } /** @@ -395,7 +396,7 @@ protected static void logWithIterator(Iterator it) { while (it.hasNext()) { String name = it.next(); Metric metric = Metric.getGlobalMetric(name); - metricsLogger.debug(metric.getName() + ":" + metric.getLogString()); + metricsLogger.debug("{}:{}", metric.getName(), metric.getLogString()); } } @@ -432,7 +433,7 @@ public boolean hasNext() { @Override public void remove() { - throw new UnsupportedOperationException("this is a read-only iterator"); + throw new UnsupportedOperationException("This is a read-only iterator"); } @Override diff --git a/src/main/java/gov/nasa/ziggy/metrics/MetricsCrud.java b/src/main/java/gov/nasa/ziggy/metrics/MetricsCrud.java index 9345983..2ee58a5 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/MetricsCrud.java +++ b/src/main/java/gov/nasa/ziggy/metrics/MetricsCrud.java @@ -20,8 +20,7 @@ public List retrieveAllMetricTypes() { return list(createZiggyQuery(MetricType.class)); } - public List metricValues(MetricType metricType, Date start, - Date end) { + public List metricValues(MetricType metricType, Date start, Date end) { ZiggyQuery query = createZiggyQuery(MetricValue.class); query.column(MetricValue_.timestamp).between(start, end).ascendingOrder(); query.column(MetricValue_.metricType).in(metricType); @@ -42,7 +41,7 @@ public TimeRange getTimestampRange(MetricType metricType) { } public long deleteOldMetrics(int maxRows) { - log.info("Preparing to delete old rows from PI_METRIC_VALUE. maxRows = " + maxRows); + log.info("Preparing to delete old rows from PI_METRIC_VALUE, maxRows={}", maxRows); long rowCount = 0; long numRowsOverLimit = 0; @@ -52,10 +51,10 @@ public long deleteOldMetrics(int maxRows) { rowCount = retrieveMetricValueRowCount(); numRowsOverLimit = rowCount - maxRows; - log.info("rowCount = " + rowCount); + log.info("rowCount={}", rowCount); if (numRowsOverLimit > 0) { - log.info("numRowsOverLimit = " + numRowsOverLimit); + log.info("numRowsOverLimit={}", numRowsOverLimit); long minId = retrieveMinimumId(); long idToDelete = minId + numRowsOverLimit - 1; @@ -65,8 +64,7 @@ public long deleteOldMetrics(int maxRows) { query.where(builder.lessThanOrEqualTo(root.get("id"), idToDelete)); int numUpdatedThisChunk = executeUpdate(query); - log.info( - "deleted " + numUpdatedThisChunk + " rows (where id <= " + idToDelete + ")"); + log.info("Deleted {} rows (where id <= {})", numUpdatedThisChunk, idToDelete); numUpdated += numUpdatedThisChunk; } else { @@ -74,7 +72,7 @@ public long deleteOldMetrics(int maxRows) { } } while (numRowsOverLimit > 0); - log.info("deleted a total of " + numUpdated + " rows."); + log.info("Deleted a total of {} rows.", numUpdated); return numUpdated; } diff --git a/src/main/java/gov/nasa/ziggy/metrics/MetricsDumper.java b/src/main/java/gov/nasa/ziggy/metrics/MetricsDumper.java index b4d6c8d..84ce652 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/MetricsDumper.java +++ b/src/main/java/gov/nasa/ziggy/metrics/MetricsDumper.java @@ -86,7 +86,8 @@ private void openFile(FileReuseMode reuseMode) { FileOutputStream fout = new FileOutputStream(metricsFile, true /* append mode */); BufferedOutputStream bout = new BufferedOutputStream(fout, BUF_SIZE_BYTES); countOut = new CountingOutputStream(bout); - printWriter = new PrintWriter(new OutputStreamWriter(countOut, ZiggyFileUtils.ZIGGY_CHARSET)); + printWriter = new PrintWriter( + new OutputStreamWriter(countOut, ZiggyFileUtils.ZIGGY_CHARSET)); } catch (IOException e) { throw new UncheckedIOException("IOException occurred on file " + metricsFile.toString(), e); diff --git a/src/main/java/gov/nasa/ziggy/metrics/MetricsOperations.java b/src/main/java/gov/nasa/ziggy/metrics/MetricsOperations.java index d37ac23..6379fe8 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/MetricsOperations.java +++ b/src/main/java/gov/nasa/ziggy/metrics/MetricsOperations.java @@ -23,8 +23,7 @@ public List metricTypes() { return performTransaction(() -> metricsCrud().retrieveAllMetricTypes()); } - public List metricValues(MetricType metricType, Date start, - Date end) { + public List metricValues(MetricType metricType, Date start, Date end) { return performTransaction(() -> metricsCrud().metricValues(metricType, start, end)); } diff --git a/src/main/java/gov/nasa/ziggy/metrics/TaskMetrics.java b/src/main/java/gov/nasa/ziggy/metrics/TaskMetrics.java index 8b9059b..755aca7 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/TaskMetrics.java +++ b/src/main/java/gov/nasa/ziggy/metrics/TaskMetrics.java @@ -5,10 +5,9 @@ import java.util.Map; import java.util.Objects; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; -import gov.nasa.ziggy.util.dispmod.DisplayModel; /** * Compute the time spent on the specified category for a list of tasks and the percentage of the @@ -18,13 +17,13 @@ */ public class TaskMetrics { private final Map categoryMetrics = new HashMap<>(); - private final List pipelineTasks; + private final List pipelineTasks; private TimeAndPercentile unallocatedTime = null; private long totalProcessingTimeMillis; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); - public TaskMetrics(List tasks) { - pipelineTasks = tasks; + public TaskMetrics(List taskListForModule) { + pipelineTasks = taskListForModule; } public void calculate() { @@ -32,13 +31,11 @@ public void calculate() { Map allocatedTimeByCategory = new HashMap<>(); if (pipelineTasks != null) { - for (PipelineTask task : pipelineTasks) { - totalProcessingTimeMillis += DisplayModel.getProcessingMillis( - task.getStartProcessingTime(), task.getEndProcessingTime()); + for (PipelineTaskDisplayData task : pipelineTasks) { + totalProcessingTimeMillis += task.getExecutionClock().totalExecutionTime(); - List summaryMetrics = pipelineTaskOperations() - .summaryMetrics(task); - for (PipelineTaskMetrics metrics : summaryMetrics) { + List pipelineTaskMetrics = task.getPipelineTaskMetrics(); + for (PipelineTaskMetric metrics : pipelineTaskMetrics) { String category = metrics.getCategory(); Long categoryTimeMillis = allocatedTimeByCategory.get(category); if (categoryTimeMillis == null) { @@ -81,6 +78,10 @@ public long getTotalProcessingTimeMillis() { return totalProcessingTimeMillis; } + PipelineTaskOperations pipelineTaskOperations() { + return pipelineTaskOperations; + } + @Override public int hashCode() { return Objects.hash(categoryMetrics, totalProcessingTimeMillis, unallocatedTime); @@ -99,8 +100,4 @@ public boolean equals(Object obj) { && totalProcessingTimeMillis == other.totalProcessingTimeMillis && Objects.equals(unallocatedTime, other.unallocatedTime); } - - PipelineTaskOperations pipelineTaskOperations() { - return pipelineTaskOperations; - } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/TimeAndPercentile.java b/src/main/java/gov/nasa/ziggy/metrics/TimeAndPercentile.java index 858cb2a..60ba9e2 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/TimeAndPercentile.java +++ b/src/main/java/gov/nasa/ziggy/metrics/TimeAndPercentile.java @@ -4,6 +4,22 @@ public class TimeAndPercentile { + private final long timeMillis; + private final double percent; + + public TimeAndPercentile(long timeMillis, double percent) { + this.timeMillis = timeMillis; + this.percent = percent; + } + + public long getTimeMillis() { + return timeMillis; + } + + public double getPercent() { + return percent; + } + @Override public int hashCode() { return Objects.hash(percent, timeMillis); @@ -21,20 +37,4 @@ public boolean equals(Object obj) { return Double.doubleToLongBits(percent) == Double.doubleToLongBits(other.percent) && timeMillis == other.timeMillis; } - - private final long timeMillis; - private final double percent; - - public TimeAndPercentile(long timeMillis, double percent) { - this.timeMillis = timeMillis; - this.percent = percent; - } - - public long getTimeMillis() { - return timeMillis; - } - - public double getPercent() { - return percent; - } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/AppendixReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/AppendixReport.java index 7ba957f..f984f36 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/AppendixReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/AppendixReport.java @@ -6,12 +6,15 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.util.dispmod.TasksDisplayModel; public class AppendixReport extends Report { - private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); + private final PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); public AppendixReport(PdfRenderer pdfRenderer) { super(pdfRenderer); @@ -20,8 +23,10 @@ public AppendixReport(PdfRenderer pdfRenderer) { public void generateReport(PipelineInstance instance, List nodes) { List tasks = pipelineInstanceNodeOperations().pipelineTasks(nodes); + List taskData = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(tasks); - TasksDisplayModel tasksDisplayModel = new TasksDisplayModel(tasks); + TasksDisplayModel tasksDisplayModel = new TasksDisplayModel(taskData); float[] colsWidth = { 0.5f, 0.5f, 2f, 1f, 1f, 0.5f, 1.5f }; printDisplayModel("Appendix A: All Tasks", tasksDisplayModel, colsWidth); @@ -37,4 +42,8 @@ public void generateReport(PipelineInstance instance, List PipelineInstanceNodeOperations pipelineInstanceNodeOperations() { return pipelineInstanceNodeOperations; } + + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/InstanceMetricsReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/InstanceMetricsReport.java index db751c6..8093e30 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/InstanceMetricsReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/InstanceMetricsReport.java @@ -46,7 +46,7 @@ public class InstanceMetricsReport { /** * Top-level directory that contains the task files. Assumes that this directory contains all of - * the task directories, which in turn contain all of the sub-task directories + * the task directories, which in turn contain all of the subtask directories */ private File rootDirectory = null; @@ -66,9 +66,9 @@ public class InstanceMetricsReport { private final TopNList instanceTopNList = new TopNList(TOP_N_INSTANCE); // Map> - Complete list of exec times, by sky group - private final Map> subTaskExecTimesByTask = new HashMap<>(); + private final Map> subtaskExecTimesByTask = new HashMap<>(); - private final List subTaskExecTimes = new ArrayList<>(200000); + private final List subtaskExecTimes = new ArrayList<>(200000); private PdfRenderer instancePdfRenderer; private PdfRenderer taskPdfRenderer; @@ -76,7 +76,7 @@ public class InstanceMetricsReport { public InstanceMetricsReport(File rootDirectory) { if (rootDirectory == null || !rootDirectory.isDirectory()) { throw new IllegalArgumentException( - "rootDirectory does not exist or is not a directory: " + rootDirectory); + "rootDirectory " + rootDirectory + " does not exist or is not a directory "); } this.rootDirectory = rootDirectory; } @@ -94,13 +94,13 @@ public void generateReport() { parseFiles(); - log.info("Instance Metrics"); + log.info("Instance metrics"); dump(instanceMetrics); - dumpTopTen(instancePdfRenderer, "Top " + TOP_N_INSTANCE + " for instance: ", + dumpTopTen(instancePdfRenderer, "Top " + TOP_N_INSTANCE + " for instance ", instanceTopNList); - JFreeChart histogram = generateHistogram("instance", subTaskExecTimes); + JFreeChart histogram = generateHistogram("instance", subtaskExecTimes); if (histogram != null) { // chart2Png(histogram, @@ -124,7 +124,7 @@ private void parseFiles() { .listFiles((FileFilter) f -> f.getName().contains("-matlab-") && f.isDirectory()); for (File taskDir : taskDirs) { - log.info("Processing: " + taskDir); + log.info("Processing {}", taskDir); String taskDirName = taskDir.getName(); Map taskMetrics = taskMetricsMap.get(taskDirName); @@ -140,19 +140,19 @@ private void parseFiles() { && pathname.isDirectory()); if (subtaskDirs == null) { - log.info("No sub-task directories found"); + log.info("No subtask directories found"); return; } - log.info("Found " + subtaskDirs.length + " sub-task directories"); + log.info("Found {} subtask directories", subtaskDirs.length); - for (File subTaskDir : subtaskDirs) { - File subTaskMetricsFile = new File(subTaskDir, METRICS_FILE_NAME); + for (File subtaskDir : subtaskDirs) { + File subtaskMetricsFile = new File(subtaskDir, METRICS_FILE_NAME); - if (subTaskMetricsFile.exists()) { - Map subTaskMetrics = Metric - .loadMetricsFromSerializedFile(subTaskMetricsFile); + if (subtaskMetricsFile.exists()) { + Map subtaskMetrics = Metric + .loadMetricsFromSerializedFile(subtaskMetricsFile); - for (Metric metric : subTaskMetrics.values()) { + for (Metric metric : subtaskMetrics.values()) { // merge this metric into the instance metrics merge(metric, instanceMetrics); // merge this metric into the task metrics @@ -162,21 +162,21 @@ private void parseFiles() { IntervalMetric totalTimeMetric = (IntervalMetric) metric; int execTime = (int) totalTimeMetric.getAverage(); instanceTopNList.add(execTime, - taskDirName + "/" + subTaskDir.getName()); - taskTopNList.add(execTime, taskDirName + "/" + subTaskDir.getName()); + taskDirName + "/" + subtaskDir.getName()); + taskTopNList.add(execTime, taskDirName + "/" + subtaskDir.getName()); addExecTime(taskDirName, totalTimeMetric.getAverage()); } } } else { - log.warn("No metrics file found in: " + subTaskDir); + log.warn("No metrics file found in {}", subtaskDir); } } - log.info("Metrics for: " + taskDirName); - dumpTopTen(taskPdfRenderer, "Top " + TOP_N_TASKS + " for task: " + taskDirName, + log.info("Metrics for {}", taskDirName); + dumpTopTen(taskPdfRenderer, "Top " + TOP_N_TASKS + " for task " + taskDirName, taskTopNList); - List taskExecTimes = subTaskExecTimesByTask.get(taskDirName); + List taskExecTimes = subtaskExecTimesByTask.get(taskDirName); JFreeChart histogram = generateHistogram(taskDirName, taskExecTimes); if (histogram != null) { @@ -191,15 +191,15 @@ private void parseFiles() { } private void addExecTime(String taskDirName, double execTime) { - List timesForTask = subTaskExecTimesByTask.get(taskDirName); + List timesForTask = subtaskExecTimesByTask.get(taskDirName); if (timesForTask == null) { timesForTask = new ArrayList<>(5000); - subTaskExecTimesByTask.put(taskDirName, timesForTask); + subtaskExecTimesByTask.put(taskDirName, timesForTask); } double timeHours = execTime / 1000.0 / 3600.0; // convert to hours timesForTask.add(timeHours); - subTaskExecTimes.add(timeHours); + subtaskExecTimes.add(timeHours); } private double[] listToArray(List list) { @@ -229,7 +229,7 @@ private JFreeChart generateHistogram(String label, List execTimes) { dataset.addSeries("execTime", values, NUM_BINS); JFreeChart chart = ChartFactory.createHistogram("Algorithm Run-time (" + label + ")", - "execTime (hours)", "Number of Sub-tasks", dataset, PlotOrientation.VERTICAL, true, + "execTime (hours)", "Number of Subtasks", dataset, PlotOrientation.VERTICAL, true, true, false); XYPlot plot = (XYPlot) chart.getPlot(); plot.setDomainPannable(true); @@ -249,10 +249,10 @@ private JFreeChart generateHistogram(String label, List execTimes) { private JFreeChart generateBoxAndWhiskers() { DefaultBoxAndWhiskerCategoryDataset dataset = new DefaultBoxAndWhiskerCategoryDataset(); - Set taskNames = subTaskExecTimesByTask.keySet(); + Set taskNames = subtaskExecTimesByTask.keySet(); for (String taskName : taskNames) { - log.info("taskDirName = " + taskName); - List execTimesForTask = subTaskExecTimesByTask.get(taskName); + log.info("taskDirName={}", taskName); + List execTimesForTask = subtaskExecTimesByTask.get(taskName); dataset.add(BoxAndWhiskerCalculator.calculateBoxAndWhiskerStatistics(execTimesForTask), taskName, taskName); } @@ -279,7 +279,7 @@ private void merge(Metric metricToMerge, Map mergeDestination) { private void dump(Map metrics) { for (Metric metric : metrics.values()) { - log.info(metric.getName() + ": " + metric.toString()); + log.info("{}: {}", metric.getName(), metric.toString()); } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/InstanceReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/InstanceReport.java index bc5d9ed..5f636a1 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/InstanceReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/InstanceReport.java @@ -13,8 +13,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; -import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.util.dispmod.InstancesDisplayModel; import gov.nasa.ziggy.util.dispmod.TaskSummaryDisplayModel; @@ -22,8 +23,9 @@ public class InstanceReport extends Report { private static final Logger log = LoggerFactory.getLogger(InstanceReport.class); - private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); public InstanceReport(PdfRenderer pdfRenderer) { super(pdfRenderer); @@ -48,28 +50,23 @@ public void generateReport(PipelineInstance instance, List timeTable.setWidthPercentage(100); addCell(timeTable, "Start", true); - addCell(timeTable, "End", true); addCell(timeTable, "Total", true); - addCell(timeTable, dateToDateString(instance.getStartProcessingTime()), false); - addCell(timeTable, dateToDateString(instance.getEndProcessingTime()), false); - String elapsedTime = instance.elapsedTime(); - addCell(timeTable, elapsedTime, false); + addCell(timeTable, dateToDateString(instance.getCreated()), false); + addCell(timeTable, instance.getExecutionClock().toString(), false); pdfRenderer.add(timeTable); pdfRenderer.println(); // Instance Summary - InstancesDisplayModel instancesDisplayModel = new InstancesDisplayModel(instance); - printDisplayModel("", instancesDisplayModel); + printDisplayModel("", new InstancesDisplayModel(instance)); pdfRenderer.println(); // Task Summary - TaskSummaryDisplayModel tasksDisplayModel = new TaskSummaryDisplayModel( - pipelineInstanceNodeOperations().taskCounts(nodes)); - printDisplayModel("Pipeline Task Summary", tasksDisplayModel); + printDisplayModel("Pipeline Task Summary", + new TaskSummaryDisplayModel(pipelineTaskDisplayDataOperations().taskCounts(nodes))); pdfRenderer.println(); @@ -123,9 +120,10 @@ private void transfersForNode(PdfPTable transfersTable, PipelineInstanceNode nod List tasks = pipelineTaskOperations().allPipelineTasks(); for (PipelineTask task : tasks) { - List taskMetrics = pipelineTaskOperations().summaryMetrics(task); + List taskMetrics = pipelineTaskDataOperations() + .pipelineTaskMetrics(task); - for (PipelineTaskMetrics taskMetric : taskMetrics) { + for (PipelineTaskMetric taskMetric : taskMetrics) { if (taskMetric.getCategory().equals(sizeCategory)) { sizeStats.addValue(taskMetric.getValue()); } else if (taskMetric.getCategory().equals(timeCategory)) { @@ -139,9 +137,9 @@ private void transfersForNode(PdfPTable transfersTable, PipelineInstanceNode nod double bytesPerSecondForNode = bytesForNode / (millisForNode / 1000); - log.info("bytesForNode = " + bytesForNode); - log.info("millisForNode = " + millisForNode); - log.info("bytesPerSecondForNode = " + bytesPerSecondForNode); + log.info("bytesForNode = {}", bytesForNode); + log.info("millisForNode = {}", millisForNode); + log.info("bytesPerSecondForNode = {}", bytesPerSecondForNode); addCell(transfersTable, node.getModuleName()); addCell(transfersTable, label); @@ -149,11 +147,15 @@ private void transfersForNode(PdfPTable transfersTable, PipelineInstanceNode nod addCell(transfersTable, rateFormatter.format(bytesPerSecondForNode)); } - private PipelineInstanceNodeOperations pipelineInstanceNodeOperations() { - return pipelineInstanceNodeOperations; - } - private PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + private PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/MatlabMetrics.java b/src/main/java/gov/nasa/ziggy/metrics/report/MatlabMetrics.java index 488dd9e..420f02b 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/MatlabMetrics.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/MatlabMetrics.java @@ -68,7 +68,7 @@ public void parseFiles() { topTen = new TopNList(10); if (cacheFile.exists()) { - log.info("Found cache file"); + log.info("Found cache file {}", cacheFile); try (ObjectInputStream ois = new ObjectInputStream( new BufferedInputStream(new FileInputStream(cacheFile)))) { CacheContents cacheContents = (CacheContents) ois.readObject(); @@ -84,34 +84,34 @@ public void parseFiles() { && f.isDirectory()); for (File taskDir : taskDirs) { - log.info("Processing: " + taskDir); + log.info("Processing {}", taskDir); SubtaskDirectoryIterator directoryIterator = new SubtaskDirectoryIterator( taskDir); if (directoryIterator.hasNext()) { - log.info("Found " + directoryIterator.numSubTasks() - + " sub-task directories"); + log.info("Found {} subtask directories", + directoryIterator.numSubtasks()); } else { - log.info("No sub-task directories found"); + log.info("No subtask directories found"); } while (directoryIterator.hasNext()) { - File subTaskDir = directoryIterator.next().getSubtaskDir(); + File subtaskDir = directoryIterator.next().getSubtaskDir(); - log.debug("STM: " + subTaskDir); + log.debug("STM {}", subtaskDir); - File subTaskMetricsFile = new File(subTaskDir, MATLAB_METRICS_FILENAME); + File subtaskMetricsFile = new File(subtaskDir, MATLAB_METRICS_FILENAME); - if (subTaskMetricsFile.exists()) { - Map subTaskMetrics = Metric - .loadMetricsFromSerializedFile(subTaskMetricsFile); + if (subtaskMetricsFile.exists()) { + Map subtaskMetrics = Metric + .loadMetricsFromSerializedFile(subtaskMetricsFile); - for (String metricName : subTaskMetrics.keySet()) { + for (String metricName : subtaskMetrics.keySet()) { if (!metricName.equals(MATLAB_CONTROLLER_EXEC_TIME_METRIC)) { - Metric metric = subTaskMetrics.get(metricName); + Metric metric = subtaskMetrics.get(metricName); - log.debug("STM: " + metricName + ": " + metric.toString()); + log.debug("STM {}={}", metricName, metric.toString()); DescriptiveStatistics metricStats = functionStats .get(metricName); @@ -125,22 +125,22 @@ public void parseFiles() { } } - Metric metric = subTaskMetrics + Metric metric = subtaskMetrics .get(MATLAB_CONTROLLER_EXEC_TIME_METRIC); if (metric != null) { - String subTaskName = subTaskDir.getParentFile().getName() + "/" - + subTaskDir.getName(); + String subtaskName = subtaskDir.getParentFile().getName() + "/" + + subtaskDir.getName(); IntervalMetric totalTimeMetric = (IntervalMetric) metric; double mean = totalTimeMetric.getAverage(); totalTimeStats.addValue(mean); - topTen.add((long) mean, subTaskName); + topTen.add((long) mean, subtaskName); } else { - log.warn("no metric found with name: " - + MATLAB_CONTROLLER_EXEC_TIME_METRIC + " in:" + subTaskDir); + log.warn("No metric found with name {} in {}", + MATLAB_CONTROLLER_EXEC_TIME_METRIC, subtaskDir); } } else { - log.warn("No metrics file found in: " + subTaskDir); + log.warn("No metrics file found in {}", subtaskDir); } } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/MatlabReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/MatlabReport.java index b9f88a6..6defd44 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/MatlabReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/MatlabReport.java @@ -47,12 +47,12 @@ private void generateExecTimeReport() { DefaultPieDataset functionBreakdownDataset = new DefaultPieDataset(); - log.info("breakdown report"); + log.info("Breakdown report"); for (String metricName : matlabFunctionStats.keySet()) { String label = shortMetricName(metricName); - log.info("processing metric: " + label); + log.info("Processing metric {}", label); DescriptiveStatistics functionStats = matlabFunctionStats.get(metricName); double functionTime = functionStats.getSum(); @@ -75,7 +75,7 @@ private void generateExecTimeReport() { HumanReadableStatistics values = millisToHumanReadable(matlabStats); JFreeChart execHistogram = generateHistogram("MATLAB Controller Run Time", - "Time (" + values.getUnit() + ")", "Sub-Tasks", values.getValues(), 100); + "Time (" + values.getUnit() + ")", "Subtasks", values.getValues(), 100); if (execHistogram != null) { pdfRenderer.printChart(execHistogram, CHART3_WIDTH, CHART3_HEIGHT); @@ -99,22 +99,22 @@ private void generateMemoryReport() { TopNList topTen = new TopNList(10); for (File taskDir : taskDirs) { - log.info("Processing: " + taskDir); + log.info("Processing {}", taskDir); Memdrone memdrone = new Memdrone(moduleName, instanceId); Map taskStats = memdrone.statsByPid(); - Map pidMap = memdrone.subTasksByPid(); + Map pidMap = memdrone.subtasksByPid(); Set pids = taskStats.keySet(); for (String pid : pids) { - String subTaskName = pidMap.get(pid); - if (subTaskName == null) { - subTaskName = "?:" + pid; + String subtaskName = pidMap.get(pid); + if (subtaskName == null) { + subtaskName = "?:" + pid; } double max = taskStats.get(pid).getMax(); memoryStats.addValue(max); - topTen.add((long) max, subTaskName); + topTen.add((long) max, subtaskName); } } diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/Memdrone.java b/src/main/java/gov/nasa/ziggy/metrics/report/Memdrone.java index b3ce540..ff878dd 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/Memdrone.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/Memdrone.java @@ -165,12 +165,11 @@ public void startMemdrone() { Path memdronePath = latestMemdronePath(); commandLine.addArgument(memdronePath.toString()); if (watchdogMap.containsKey(nameRoot)) { - log.info("Memdrone for module " + binaryName + ", instance " + instanceId - + " already running"); + log.info("Memdrone for module {}, instance {} already running", binaryName, instanceId); return; } - log.info("Starting memdrone for module " + binaryName + " in instance " + instanceId); + log.info("Starting memdrone for module {} in instance {}", binaryName, instanceId); ExternalProcess memdroneProcess = ExternalProcess.simpleExternalProcess(commandLine); memdroneProcess.execute(false); watchdogMap.put(nameRoot, memdroneProcess.getWatchdog()); @@ -181,13 +180,13 @@ public void startMemdrone() { */ public void stopMemdrone() { if (watchdogMap.containsKey(nameRoot)) { - log.info("Stopping memdrone for module " + binaryName + " in instance " + instanceId); + log.info("Stopping memdrone for module {} in instance {}", binaryName, instanceId); watchdogMap.get(nameRoot).destroyProcess(); log.info("Memdrone stopped"); watchdogMap.remove(nameRoot); } else { - log.info("No memdrone script was running for module " + binaryName + " in instance " - + instanceId); + log.info("No memdrone script was running for module {} in instance {}", binaryName, + instanceId); } } @@ -239,19 +238,19 @@ public Map statsByPid() { justification = SpotBugsUtils.DESERIALIZATION_JUSTIFICATION) @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - public Map subTasksByPid() { + public Map subtasksByPid() { Path cacheFile = latestMemdronePath().resolve(PID_MAP_CACHE_FILENAME); - Map pidToSubTask = null; + Map pidToSubtask = null; if (Files.exists(cacheFile)) { - log.info("Using pid cache file"); + log.info("Using PID cache file {}", cacheFile); try (ObjectInputStream ois = new ObjectInputStream( new BufferedInputStream(new FileInputStream(cacheFile.toFile())))) { @SuppressWarnings("unchecked") Map obj = (Map) ois.readObject(); - pidToSubTask = obj; + pidToSubtask = obj; - log.debug("pid cache: " + obj); + log.debug("PID cache {}", obj); } catch (IOException e) { throw new UncheckedIOException( "IOException occurred reading from " + cacheFile.toString(), e); @@ -261,11 +260,11 @@ public Map subTasksByPid() { throw new AssertionError(e); } } else { // no cache - log.info("Creating pid cache file"); - pidToSubTask = createPidMapCache(); + log.info("Creating PID cache file"); + pidToSubtask = createPidMapCache(); } - return pidToSubTask; + return pidToSubtask; } /** @@ -281,10 +280,10 @@ public Map createStatsCache() { .listFiles((FileFilter) f -> f.getName().startsWith("memdrone-") && f.isFile()); if (memdroneLogs != null) { - log.info("Number of memdrone-* files found: " + memdroneLogs.length); + log.info("Found {} memdrone-* files", memdroneLogs.length); for (File memdroneLog : memdroneLogs) { - log.info("Processing: " + memdroneLog); + log.info("Processing {}", memdroneLog); String filename = memdroneLog.getName(); String host = filename.substring(filename.indexOf("-") + 1, filename.indexOf(".")); @@ -318,9 +317,9 @@ public Map createStatsCache() { @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public Map createPidMapCache() { Path cacheFile = latestMemdronePath().resolve(PID_MAP_CACHE_FILENAME); - Map pidToSubTask = new HashMap<>(); + Map pidToSubtask = new HashMap<>(); - log.info("cacheFile: " + cacheFile); + log.info("cacheFile={}", cacheFile); Pattern taskPattern = Pattern .compile(binaryName + "-" + Long.toString(instanceId) + "-" + "\\d+"); @@ -331,22 +330,22 @@ public Map createPidMapCache() { .listFiles((FileFilter) f -> taskPattern.matcher(f.getName()).matches()); if (taskDirs != null) { for (File taskDir : taskDirs) { - log.info("Processing task " + taskDir.getName()); + log.info("Processing task {}", taskDir.getName()); Matcher taskMatcher = taskPattern.matcher(taskDir.getName()); taskMatcher.matches(); String taskDirName = taskDir.getName(); taskDirName = taskDirName.substring(taskDirName.lastIndexOf("-") + 1); // Find the subtask directories and loop over them. - File[] subTaskDirs = taskDir + File[] subtaskDirs = taskDir .listFiles((FileFilter) f -> f.getName().contains("st-") && f.isDirectory()); - if (subTaskDirs != null) { - for (File subTaskDir : subTaskDirs) { - String subTaskId = taskDirName + "/" + subTaskDir.getName(); - log.debug("processing subTaskId: " + subTaskId); + if (subtaskDirs != null) { + for (File subtaskDir : subtaskDirs) { + String subtaskId = taskDirName + "/" + subtaskDir.getName(); + log.debug("Processing subtaskId {}", subtaskId); // Get the contents of the PID filename and populate the map - File pidsFile = new File(subTaskDir, + File pidsFile = new File(subtaskDir, MatlabJavaInitialization.MATLAB_PIDS_FILENAME); if (pidsFile.exists()) { String hostPid; @@ -358,7 +357,7 @@ public Map createPidMapCache() { "Unable to read from file " + pidsFile.toString(), e); } - pidToSubTask.put(hostPid, subTaskId); + pidToSubtask.put(hostPid, subtaskId); } } } @@ -369,12 +368,12 @@ public Map createPidMapCache() { ObjectOutputStream oos = new ObjectOutputStream(new BufferedOutputStream( new FileOutputStream(cacheFile.toAbsolutePath().toString())))) { - oos.writeObject(pidToSubTask); + oos.writeObject(pidToSubtask); oos.flush(); } catch (IOException e) { throw new UncheckedIOException("Unable to write to file " + cacheFile.toString(), e); } - return pidToSubTask; + return pidToSubtask; } private Date date() { diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneLog.java b/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneLog.java index ae0c2e1..1fc5a71 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneLog.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneLog.java @@ -37,18 +37,18 @@ public class MemdroneLog { public MemdroneLog(File memdroneLogFile) { if (!memdroneLogFile.exists()) { throw new PipelineException( - "Specified memdrone file does not exist: " + memdroneLogFile); + "Specified memdrone file " + memdroneLogFile + " does not exist"); } if (!memdroneLogFile.isFile()) { throw new PipelineException( - "Specified memdrone file is not a regular file: " + memdroneLogFile); + "Specified memdrone file " + memdroneLogFile + " is not a regular file"); } try { input = new FileInputStream(memdroneLogFile); } catch (FileNotFoundException e) { - throw new UncheckedIOException("File " + memdroneLogFile.toString() + " not found", e); + throw new UncheckedIOException("File " + memdroneLogFile + " not found", e); } this.memdroneLogFile = memdroneLogFile; parse(); @@ -61,7 +61,7 @@ public MemdroneLog(InputStream input) { @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private void parse() { - log.info("Parse started"); + log.info("Parsing"); logContents = new HashMap<>(); @@ -87,9 +87,9 @@ private void parse() { "IOException occurred reading from " + memdroneLogFile.toString(), e); } - log.info("Parse complete"); - log.info("lineCount: " + lineCount); - log.info("skipCount: " + skipCount); + log.info("Parsing...done"); + log.info("lineCount = {}", lineCount); + log.info("skipCount = {}", skipCount); } public Map getLogContents() { diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneSample.java b/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneSample.java index 88cb10f..6c2654c 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneSample.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/MemdroneSample.java @@ -52,7 +52,7 @@ private boolean parse(String memdroneLogLine) { String[] elements = memdroneLogLine.split("\\s+"); if (elements.length != 9) { - log.warn("Parse error, num elements != 11 : " + memdroneLogLine); + log.warn("Parse error, {} elements != 9", memdroneLogLine); return false; } String timestampString = elements[0] + " " + // day of week @@ -68,7 +68,7 @@ private boolean parse(String memdroneLogLine) { timestampMillis = parseDate(timestampString); memoryKilobytes = Integer.parseInt(elements[7]); } catch (ParseException | NumberFormatException e) { - log.warn("Parse error: " + e); + log.warn("Parse error {}", e); return false; } return true; diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/NodeReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/NodeReport.java index c35fc66..2ac4b1d 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/NodeReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/NodeReport.java @@ -16,8 +16,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; public class NodeReport extends Report { @@ -27,6 +28,7 @@ public class NodeReport extends Report { private Map categoryTopTen; private Map categoryUnits; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); public NodeReport(PdfRenderer pdfRenderer) { super(pdfRenderer); @@ -44,11 +46,11 @@ public void generateReport(PipelineInstanceNode node) { orderedCategoryNames = new LinkedList<>(); - Map> taskMetricsByTask = pipelineTaskOperations() + Map> taskMetricsByTask = pipelineTaskDataOperations() .taskMetricsByTask(node); for (PipelineTask task : taskMetricsByTask.keySet()) { - for (PipelineTaskMetrics taskMetric : taskMetricsByTask.get(task)) { + for (PipelineTaskMetric taskMetric : taskMetricsByTask.get(task)) { String category = taskMetric.getCategory(); categoryUnits.put(category, taskMetric.getUnits()); @@ -75,7 +77,7 @@ public void generateReport(PipelineInstanceNode node) { categoryTopTen.put(category, topTen); } - topTen.add(value, "ID: " + task.getId()); + topTen.add(value, "ID: " + task); List valueList = categoryMetrics.get(category); @@ -84,21 +86,21 @@ public void generateReport(PipelineInstanceNode node) { categoryMetrics.put(category, valueList); } - valueList.add(new PipelineTaskMetricValue(task.getId(), value)); + valueList.add(new PipelineTaskMetricValue(task, value)); } } DefaultCategoryDataset categoryTaskDataset = new DefaultCategoryDataset(); - log.info("summary report"); + log.info("Summary report"); for (String category : orderedCategoryNames) { - log.info("processing category: " + category); + log.info("Processing category {}", category); if (categoryIsTime(category)) { List values = categoryMetrics.get(category); for (PipelineTaskMetricValue value : values) { - Long taskId = value.getPipelineTaskId(); + Long taskId = value.getPipelineTask().getId(); Long valueMillis = value.getMetricValue(); double valueMins = valueMillis / (1000.0 * 60); categoryTaskDataset.addValue(valueMins, category, taskId); @@ -113,7 +115,7 @@ public void generateReport(PipelineInstanceNode node) { pdfRenderer.newPage(); - // task breakdown table + // Task breakdown table pdfRenderer.printText("Wall Time Breakdown by Task and Category", PdfRenderer.h1Font); pdfRenderer.println(); @@ -179,6 +181,10 @@ PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + /** * Container for the ID of a {@link PipelineTask} and a metric value. * @@ -186,16 +192,16 @@ PipelineTaskOperations pipelineTaskOperations() { */ private static class PipelineTaskMetricValue { - private final long pipelineTaskId; + private final PipelineTask pipelineTask; private final long metricValue; - public PipelineTaskMetricValue(long pipelineTaskId, long metricValue) { - this.pipelineTaskId = pipelineTaskId; + public PipelineTaskMetricValue(PipelineTask pipelineTask, long metricValue) { + this.pipelineTask = pipelineTask; this.metricValue = metricValue; } - public long getPipelineTaskId() { - return pipelineTaskId; + public PipelineTask getPipelineTask() { + return pipelineTask; } public long getMetricValue() { diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java index dd0caf9..19a23e3 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java @@ -102,7 +102,7 @@ public Path generateReport() { "Unable to create directory " + outputPath.getParent().toString(), e); } - log.info("Writing report to {}...", outputPath.toString()); + log.info("Writing report to {}", outputPath.toString()); PdfRenderer pdfRenderer = new PdfRenderer(outputPath.toFile(), false); @@ -149,7 +149,7 @@ private List selectNodes(List instan throw new PipelineException("Invalid node range: " + nodes); } - log.info("processing nodes " + startNode + " to " + endNode); + log.info("Processing nodes {} to {}", startNode, endNode); return instanceNodes.subList(startNode, endNode + 1); } @@ -175,14 +175,14 @@ private void generateNodeReport(PdfRenderer pdfRenderer, PipelineInstanceNode no pdfRenderer.newPage(); - log.info("category report"); + log.info("Category report"); List orderedCategoryNames = nodeReport.getOrderedCategoryNames(); Map categoryStats = nodeReport.getCategoryStats(); Map topTen = nodeReport.getCategoryTopTen(); for (String category : orderedCategoryNames) { - log.info("processing category: " + category); + log.info("Processing category {}", category); boolean isTime = nodeReport.categoryIsTime(category); CategoryReport categoryReport = new CategoryReport(category, pdfRenderer, isTime); diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/Report.java b/src/main/java/gov/nasa/ziggy/metrics/report/Report.java index 42925e0..21a652a 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/Report.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/Report.java @@ -177,7 +177,7 @@ protected String formatTime(long timeMillis) { protected void generateSummaryTable(String label, DescriptiveStatistics stats, TopNList topTen, Format f) { - log.info("Generating report for: " + label); + log.info("Generating report for {}", label); PdfPTable layoutTable = new PdfPTable(2); PdfPTable statsTable = new PdfPTable(2); diff --git a/src/main/java/gov/nasa/ziggy/models/ModelImporter.java b/src/main/java/gov/nasa/ziggy/models/ModelImporter.java index 229ebc4..1757132 100644 --- a/src/main/java/gov/nasa/ziggy/models/ModelImporter.java +++ b/src/main/java/gov/nasa/ziggy/models/ModelImporter.java @@ -82,10 +82,10 @@ public ModelImporter(Path dataImportPath, String modelDescription) { */ public void importModels(List files) { - log.info("Importing models..."); + log.info("Importing models"); if (modelTypesToImport.isEmpty()) { modelTypesToImport.addAll(modelTypes()); - log.info("Retrieved " + modelTypesToImport.size() + " model types from database"); + log.info("Retrieved {} model types from database", modelTypesToImport.size()); } Map> modelFilesByModelType = new HashMap<>(); @@ -111,7 +111,7 @@ public void importModels(List files) { long unlockedModelRegistryId = mergeRegistryAndReturnUnlockedId(modelRegistry); log.info("Update of model registry complete"); log.info("Importing models...done"); - log.info("Current unlocked model registry ID == " + unlockedModelRegistryId); + log.info("Current unlocked model registry ID is {}", unlockedModelRegistryId); } /** @@ -179,8 +179,8 @@ private void addModels(ModelRegistry modelRegistry, ModelType modelType, Set modelVersions = new TreeSet<>(modelFilesByVersionId.keySet()); for (String version : modelVersions) { createModel(modelRegistry, modelType, modelDir, modelFilesByVersionId.get(version)); - log.info(modelFilesByVersionId.size() + " models of type " + modelType.getType() - + " added to datastore"); + log.info("Added {} models of type {} to datastore", modelFilesByVersionId.size(), + modelType.getType()); } } @@ -209,7 +209,7 @@ private void createModel(ModelRegistry modelRegistry, ModelType modelType, Path currentModelRegistryMetadata); modelMetadata.setDataReceiptTaskId(dataReceiptTaskId); } catch (Exception e) { - log.error("Unable to create model metadata for file " + modelFile); + log.error("Unable to create model metadata for file {}", modelFile); failedImports.add(dataImportPath.relativize(modelFile)); return; } @@ -219,7 +219,7 @@ private void createModel(ModelRegistry modelRegistry, ModelType modelType, Path try { move(modelFile, destinationFile); } catch (Exception e) { - log.error("Unable to import file " + modelFile + " into datastore"); + log.error("Unable to import file {} into datastore", modelFile); failedImports.add(dataImportPath.relativize(modelFile)); return; } @@ -227,8 +227,8 @@ private void createModel(ModelRegistry modelRegistry, ModelType modelType, Path // If all that worked, then we can update the model registry persistModelMetadata(modelMetadata); modelRegistry.updateModelMetadata(modelMetadata); - log.info("Imported file " + modelFile + " to models directory as " - + modelMetadata.getDatastoreFileName() + " of type " + modelType.getType()); + log.info("Imported file {} to models directory as {} of type {}", modelFile, + modelMetadata.getDatastoreFileName(), modelType.getType()); successfulImports.add(datastoreRoot.relativize(destinationFile)); } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java index 20daea4..6bca47f 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java @@ -1,5 +1,10 @@ package gov.nasa.ziggy.module; +import java.io.BufferedWriter; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.UncheckedIOException; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; import java.nio.file.Files; @@ -13,10 +18,13 @@ import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale;; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.ZiggyFileUtils;; /** * Superclass for algorithm execution. This includes local execution via the @@ -32,25 +40,32 @@ public abstract class AlgorithmExecutor { public static final String ZIGGY_PROGRAM = "ziggy"; protected static final String NODE_MASTER_NAME = "compute-node-master"; + public static final String ACTIVE_CORES_FILE_NAME = ".activeCoresPerNode"; + public static final String WALL_TIME_FILE_NAME = ".requestedWallTimeSeconds"; protected final PipelineTask pipelineTask; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); - private StateFile stateFile; + private PbsParameters pbsParameters; /** * Returns a new instance of the appropriate {@link AlgorithmExecutor} subclass. */ public static final AlgorithmExecutor newInstance(PipelineTask pipelineTask) { - return newInstance(pipelineTask, new PipelineTaskOperations()); + return newInstance(pipelineTask, new PipelineTaskOperations(), + new PipelineTaskDataOperations()); } /** * Version of {@link #newInstance(PipelineTask)} that accepts a user-supplied * {@link PipelineTaskOperations} instances. Allows these classes to be mocked for testing. + * + * @param pipelineTaskDataOperations2 */ static final AlgorithmExecutor newInstance(PipelineTask pipelineTask, - PipelineTaskOperations pipelineTaskOperations) { + PipelineTaskOperations pipelineTaskOperations, + PipelineTaskDataOperations pipelineTaskDataOperations) { if (pipelineTask == null) { log.debug("Pipeline task is null, returning LocalAlgorithmExecutor instance"); @@ -67,15 +82,15 @@ static final AlgorithmExecutor newInstance(PipelineTask pipelineTask, log.debug("Remote execution not selected, returning LocalAlgorithmExecutor instance"); return new LocalAlgorithmExecutor(pipelineTask); } - log.debug("Total subtasks " + pipelineTask.getTotalSubtaskCount()); - log.debug("Completed subtasks " + pipelineTask.getCompletedSubtaskCount()); - int subtasksToRun = pipelineTask.getTotalSubtaskCount() - - pipelineTask.getCompletedSubtaskCount(); + SubtaskCounts subtaskCounts = pipelineTaskDataOperations.subtaskCounts(pipelineTask); + log.debug("Total subtasks {}", subtaskCounts.getTotalSubtaskCount()); + log.debug("Completed subtasks {}", subtaskCounts.getCompletedSubtaskCount()); + int subtasksToRun = subtaskCounts.getTotalSubtaskCount() + - subtaskCounts.getCompletedSubtaskCount(); if (subtasksToRun < remoteParams.getMinSubtasksForRemoteExecution()) { - log.info("Number subtasks to run (" + subtasksToRun - + ") less than min subtasks for remote execution (" - + remoteParams.getMinSubtasksForRemoteExecution() + ")"); - log.info("Executing task " + pipelineTask.getId() + " locally"); + log.info("Number subtasks to run ({}) less than min subtasks for remote execution ({})", + subtasksToRun, remoteParams.getMinSubtasksForRemoteExecution()); + log.info("Executing task {} locally", pipelineTask); return new LocalAlgorithmExecutor(pipelineTask); } return newRemoteInstance(pipelineTask); @@ -114,45 +129,44 @@ protected AlgorithmExecutor(PipelineTask pipelineTask) { public void submitAlgorithm(TaskConfiguration inputsHandler) { prepareToSubmitAlgorithm(inputsHandler); + writeActiveCoresFile(); + writeWallTimeFile(); IntervalMetric.measure(PipelineMetrics.SEND_METRIC, () -> { - log.info("Submitting task for execution (taskId=" + pipelineTask.getId() + ")"); + log.info("Submitting task for execution (taskId={})", pipelineTask); Files.createDirectories(algorithmLogDir()); - Files.createDirectories(DirectoryProperties.stateFilesDir()); Files.createDirectories(taskDataDir()); SubtaskUtils.clearStaleAlgorithmStates( new TaskDirectoryManager(pipelineTask).taskDir().toFile()); - log.info("Start remote monitoring (taskId=" + pipelineTask.getId() + ")"); - submitForExecution(stateFile); + log.info("Start remote monitoring (taskId={})", pipelineTask); + submitForExecution(); + writeQueuedTimestampFile(); return null; }); } private void prepareToSubmitAlgorithm(TaskConfiguration inputsHandler) { - // execute the external process on a remote host - int numSubtasks; - PbsParameters pbsParameters = null; PipelineDefinitionNodeExecutionResources executionResources = pipelineTaskOperations() .executionResources(pipelineTask); - + int numSubtasks; // Initial submission: this is indicated by a non-null task configuration manager if (inputsHandler != null) { // indicates initial submission - log.info("Processing initial submission of task " + pipelineTask.getId()); + log.info("Processing initial submission of task {}", pipelineTask); numSubtasks = inputsHandler.getSubtaskCount(); + pipelineTaskDataOperations().updateSubtaskCounts(pipelineTask, numSubtasks, -1, -1); pbsParameters = generatePbsParameters(executionResources, numSubtasks); // Resubmission: this is indicated by a null task configuration manager, which // means that subtask counts are available in the database - } else - - { - log.info("Processing resubmission of task " + pipelineTask.getId()); - numSubtasks = pipelineTask.getTotalSubtaskCount(); - int numCompletedSubtasks = pipelineTask.getCompletedSubtaskCount(); + } else { + log.info("Processing resubmission of task {}", pipelineTask); + SubtaskCounts subtaskCounts = pipelineTaskDataOperations().subtaskCounts(pipelineTask); + numSubtasks = subtaskCounts.getTotalSubtaskCount(); + int numCompletedSubtasks = subtaskCounts.getCompletedSubtaskCount(); // Scale the total subtasks to get to the number that still need to be processed double subtaskCountScaleFactor = (double) (numSubtasks - numCompletedSubtasks) @@ -162,23 +176,24 @@ private void prepareToSubmitAlgorithm(TaskConfiguration inputsHandler) { pbsParameters = generatePbsParameters(executionResources, (int) (numSubtasks * subtaskCountScaleFactor)); } + } - stateFile = StateFile.generateStateFile(pipelineTask, pbsParameters, numSubtasks); + /** Writes the number of active cores per node to a file in the task directory. */ + private void writeActiveCoresFile() { + writeActiveCoresFile(workingDir(), activeCores()); } - /** - * Resubmit the pipeline task to the appropriate {@link AlgorithmMonitor}. This is used in the - * case where the supervisor has been stopped and restarted but tasks are still running (usually - * remotely). This notifies the monitor that there are tasks that it should be looking out for. - */ - public void resumeMonitoring() { - prepareToSubmitAlgorithm(null); - addToMonitor(stateFile); + private void writeWallTimeFile() { + writeWallTimeFile(workingDir(), wallTime()); } - protected abstract void addToMonitor(StateFile stateFile); + protected abstract void addToMonitor(); - protected abstract void submitForExecution(StateFile stateFile); + protected abstract void submitForExecution(); + + protected abstract String activeCores(); + + protected abstract String wallTime(); /** * Generates an updated instance of {@link PbsParameters}. The method is abstract because each @@ -200,22 +215,47 @@ protected Path workingDir() { return taskDataDir().resolve(pipelineTask.taskBaseName()); } + protected void writeQueuedTimestampFile() { + TimestampFile.create(workingDir().toFile(), TimestampFile.Event.QUEUED); + } + public abstract AlgorithmType algorithmType(); protected PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } - public StateFile getStateFile() { - return stateFile; + public PbsParameters getPbsParameters() { + return pbsParameters; } - public enum AlgorithmType { - /** local execution only */ - LOCAL, + protected PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } - /** Pleiades execution (database server is inside NAS enclave) */ - REMOTE + // Broken out for use in ComputeNodeMaster unit tests. + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + static void writeActiveCoresFile(Path taskDir, String activeCoresPerNode) { + try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter( + new FileOutputStream(taskDir.resolve(ACTIVE_CORES_FILE_NAME).toFile()), + ZiggyFileUtils.ZIGGY_CHARSET))) { + writer.write(activeCoresPerNode); + writer.newLine(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + // Broken out for use in ComputeNodeMaster unit tests. + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + static void writeWallTimeFile(Path taskDir, String wallTime) { + try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter( + new FileOutputStream(taskDir.resolve(WALL_TIME_FILE_NAME).toFile()), + ZiggyFileUtils.ZIGGY_CHARSET))) { + writer.write(wallTime); + writer.newLine(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } } } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java index 4caf179..f83351b 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java @@ -1,8 +1,6 @@ package gov.nasa.ziggy.module; import java.io.File; -import java.io.IOException; -import java.io.UncheckedIOException; import java.nio.file.Path; import org.slf4j.Logger; @@ -55,17 +53,7 @@ public void doPostProcessing() { @Override @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public File getTaskDir(boolean cleanExisting) { - File taskDir = allocateTaskDir(cleanExisting); - if (isRemote()) { - File stateFileLockFile = new File(taskDir, StateFile.LOCK_FILE_NAME); - try { - stateFileLockFile.createNewFile(); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to create file " + stateFileLockFile.toString(), e); - } - } - return taskDir; + return allocateTaskDir(cleanExisting); } /* @@ -75,7 +63,7 @@ public File getTaskDir(boolean cleanExisting) { */ @Override public boolean isRemote() { - return executor.algorithmType() == AlgorithmExecutor.AlgorithmType.REMOTE; + return executor.algorithmType() == AlgorithmType.REMOTE; } @Override @@ -98,7 +86,7 @@ private File allocateTaskDir(boolean cleanExisting) { if (taskDir == null) { taskDir = taskDirManager.allocateTaskDir(cleanExisting); - log.info("defaultWorkingDir = " + taskDir); + log.info("defaultWorkingDir={}", taskDir); } return taskDir.toFile(); } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java index 339756a..5fee01a 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java @@ -1,33 +1,37 @@ package gov.nasa.ziggy.module; -import java.io.File; +import java.nio.file.Files; +import java.nio.file.Paths; import java.util.ArrayList; -import java.util.Arrays; import java.util.Collection; -import java.util.LinkedList; +import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ScheduledThreadPoolExecutor; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; +import org.apache.commons.collections4.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; +import gov.nasa.ziggy.module.remote.PbsLogParser; +import gov.nasa.ziggy.module.remote.QueueCommandManager; +import gov.nasa.ziggy.module.remote.RemoteJobInformation; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; -import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.messages.AllJobsFinishedMessage; import gov.nasa.ziggy.services.messages.MonitorAlgorithmRequest; -import gov.nasa.ziggy.services.messages.WorkerStatusMessage; +import gov.nasa.ziggy.services.messages.TaskProcessingCompleteMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.supervisor.PipelineSupervisor; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -36,58 +40,61 @@ /** * Monitors algorithm processing by monitoring state files. *

- * Two instances are used: one for local tasks and the other for remote execution jobs (on HPC - * and/or cloud systems). Each one checks at regular intervals for updates to the {@link StateFile} - * for the specific task. The intervals are managed by a {@link ScheduledThreadPoolExecutor}. + * Each task has an assigned {@link TaskMonitor} instance that periodically counts the states of + * subtasks to determine the overall progress of that task. In addition, there is a periodic check + * of PBS log files that allows the monitor to determine whether some or all of the remote jobs for + * a given task have failed. * * @author Todd Klaus * @author PT + * @author Bill Wohler */ public class AlgorithmMonitor implements Runnable { private static final Logger log = LoggerFactory.getLogger(AlgorithmMonitor.class); - private static final long SSH_POLL_INTERVAL_MILLIS = 10 * 1000; // 10 secs + private static final long REMOTE_POLL_INTERVAL_MILLIS = 10 * 1000; // 10 secs private static final long LOCAL_POLL_INTERVAL_MILLIS = 2 * 1000; // 2 seconds + private static final long FINISHED_JOBS_POLL_INTERVAL_MILLIS = 10 * 1000; - private static AlgorithmMonitor localMonitoringInstance = null; - private static AlgorithmMonitor remoteMonitoringInstance = null; - private static ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(2); + private ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(1); - private List corruptedStateFileNames = new ArrayList<>(); private boolean startLogMessageWritten = false; - private AlgorithmType algorithmType; - private String monitorVersion; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); private PipelineExecutor pipelineExecutor = new PipelineExecutor(); + private PbsLogParser pbsLogParser = new PbsLogParser(); + private QueueCommandManager queueCommandManager = QueueCommandManager.newInstance(); - JobMonitor jobMonitor = null; + private final Map> jobsInformationByTask = new ConcurrentHashMap<>(); + private final Map taskMonitorByTask = new ConcurrentHashMap<>(); + private AllJobsFinishedMessage allJobsFinishedMessage; - private final ConcurrentHashMap state = new ConcurrentHashMap<>(); + // For testing only. + private Disposition disposition; /** What needs to be done after a task exits the state file checks loop: */ - private enum Disposition { + enum Disposition { // Algorithm processing is complete. Persist results. PERSIST { @Override public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) { - StateFile stateFile = new StateFile(pipelineTask.getModuleName(), - pipelineTask.getPipelineInstanceId(), pipelineTask.getId()) - .newStateFileFromDiskFile(); - if (stateFile.getNumFailed() != 0) { + SubtaskCounts subtaskCounts = monitor.pipelineTaskDataOperations() + .subtaskCounts(pipelineTask); + if (subtaskCounts.getFailedSubtaskCount() != 0) { log.warn("{} subtasks out of {} failed but task completed", - stateFile.getNumFailed(), stateFile.getNumComplete()); + subtaskCounts.getFailedSubtaskCount(), + subtaskCounts.getTotalSubtaskCount()); monitor.alertService() - .generateAndBroadcastAlert("Algorithm Monitor", pipelineTask.getId(), + .generateAndBroadcastAlert("Algorithm Monitor", pipelineTask, Severity.WARNING, "Failed subtasks, see logs for details"); } + if (monitor.jobsInformationByTask.containsKey(pipelineTask)) { + monitor.jobsInformationByTask.remove(pipelineTask); + } + log.info("Sending task {} to worker to persist results", pipelineTask); - log.info("Sending task with id: " + pipelineTask.getId() - + " to worker to persist results"); - - monitor.pipelineExecutor() - .persistTaskResults( - monitor.pipelineTaskOperations().pipelineTask(pipelineTask.getId())); + monitor.pipelineExecutor().persistTaskResults(pipelineTask); } }, @@ -95,17 +102,19 @@ public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) RESUBMIT { @Override public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) { - log.warn("Resubmitting task with id: " + pipelineTask.getId() - + " for additional processing"); + log.warn("Resubmitting task {} for additional processing", pipelineTask); monitor.alertService() - .generateAndBroadcastAlert("Algorithm Monitor", pipelineTask.getId(), - Severity.WARNING, "Resubmitting task for further processing"); - PipelineTask databaseTask = monitor.pipelineTaskOperations() - .prepareTaskForAutoResubmit(pipelineTask); + .generateAndBroadcastAlert("Algorithm Monitor", pipelineTask, Severity.WARNING, + "Resubmitting task for further processing"); + monitor.pipelineTaskDataOperations().prepareTaskForAutoResubmit(pipelineTask); + + if (monitor.jobsInformationByTask.containsKey(pipelineTask)) { + monitor.jobsInformationByTask.remove(pipelineTask); + } // Submit tasks for resubmission at highest priority. monitor.pipelineExecutor() - .restartFailedTasks(List.of(databaseTask), false, RunMode.RESUBMIT); + .restartFailedTasks(List.of(pipelineTask), false, RunMode.RESUBMIT); } }, @@ -114,10 +123,15 @@ public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) FAIL { @Override public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) { - log.error("Task with id " + pipelineTask.getId() + " failed on remote system, " - + "marking task as errored and not restarting."); - monitor.handleFailedTask(monitor.state.get(StateFile.invariantPart(pipelineTask))); - monitor.pipelineTaskOperations().taskErrored(pipelineTask); + log.error( + "Task {} failed on remote system, marking task as errored and not restarting", + pipelineTask); + + monitor.handleFailedTask(pipelineTask); + monitor.pipelineTaskDataOperations().taskErrored(pipelineTask); + if (monitor.jobsInformationByTask.containsKey(pipelineTask)) { + monitor.jobsInformationByTask.remove(pipelineTask); + } } }; @@ -127,178 +141,144 @@ public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) public abstract void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask); } - public static void initialize() { - synchronized (AlgorithmMonitor.class) { - if (localMonitoringInstance == null || remoteMonitoringInstance == null) { - ZiggyMessenger.subscribe(MonitorAlgorithmRequest.class, message -> { - if (message.getAlgorithmType().equals(AlgorithmType.LOCAL)) { - AlgorithmMonitor.startLocalMonitoring( - message.getStateFile().newStateFileFromDiskFile()); - } else { - AlgorithmMonitor.startRemoteMonitoring( - message.getStateFile().newStateFileFromDiskFile()); - } - }); - } - if (localMonitoringInstance == null) { - localMonitoringInstance = new AlgorithmMonitor(AlgorithmType.LOCAL); - localMonitoringInstance.startMonitoringThread(); - } - if (remoteMonitoringInstance == null) { - remoteMonitoringInstance = new AlgorithmMonitor(AlgorithmType.REMOTE); - remoteMonitoringInstance.startMonitoringThread(); - } - - // Whenever a worker sends a "last message," run an unscheduled - // update of the monitors. - ZiggyMessenger.subscribe(WorkerStatusMessage.class, message -> { - if (message.isLastMessageFromWorker()) { - localMonitoringInstance.run(); - remoteMonitoringInstance.run(); - } - }); - } + public AlgorithmMonitor() { + ZiggyMessenger.subscribe(MonitorAlgorithmRequest.class, message -> { + addToMonitor(message); + }); + ZiggyMessenger.subscribe(TaskProcessingCompleteMessage.class, message -> { + endTaskMonitoring(message.getPipelineTask()); + }); + startMonitoringThread(); } /** - * Returns the collection of {@link StateFile} instances currently being tracked by the remote - * execution monitor. + * Start the monitoring thread. */ - public static Collection remoteTaskStateFiles() { - if (remoteMonitoringInstance == null || remoteMonitoringInstance.state.isEmpty()) { - return null; + private void startMonitoringThread() { + long pollingIntervalMillis = finishedJobsPollingIntervalMillis(); + if (pollingIntervalMillis > 0) { + log.info("Starting polling with {} msec interval", REMOTE_POLL_INTERVAL_MILLIS); + threadPool.scheduleWithFixedDelay(this, 0, REMOTE_POLL_INTERVAL_MILLIS, + TimeUnit.MILLISECONDS); } - return remoteMonitoringInstance.state.values(); } - /** - * Constructor. Default scope for use in unit tests. - */ - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - AlgorithmMonitor(AlgorithmType algorithmType) { - this.algorithmType = algorithmType; - monitorVersion = algorithmType.name().toLowerCase(); - - log.info("Starting new monitor for: " + DirectoryProperties.stateFilesDir().toString()); - initializeJobMonitor(); + // Protected access for unit tests. + protected final void addToMonitor(MonitorAlgorithmRequest request) { + List remoteJobsInformation = request.getRemoteJobsInformation(); + log.info("Starting algorithm monitoring for task {}", request.getPipelineTask()); + if (!CollectionUtils.isEmpty(remoteJobsInformation)) { + jobsInformationByTask.put(request.getPipelineTask(), remoteJobsInformation); + } + TaskMonitor taskMonitor = taskMonitor(request); + taskMonitorByTask.put(request.getPipelineTask(), taskMonitor); + taskMonitor.startMonitoring(); } - /** - * Start the monitoring thread for a given monitor. - */ - void startMonitoringThread() { - long pollingIntervalMillis = pollingIntervalMillis(); - if (pollingIntervalMillis > 0) { - log.info("Starting polling on " + monitorVersion + " with " + pollingIntervalMillis - + " msec interval"); - threadPool.scheduleWithFixedDelay(this, 0, pollingIntervalMillis(), - TimeUnit.MILLISECONDS); + // Actions to be taken when a task monitor reports that a task is done. + // Protected access for unit tests. + protected final void endTaskMonitoring(PipelineTask pipelineTask) { + + // When the worker that persists the results exits, it will cause + // this method to execute even though the algorithm is no longer under + // monitoring. In that circumstance, exit now. + if (!taskMonitorByTask.containsKey(pipelineTask)) { + return; } - } - private String username() { - String username = System.getenv("USERNAME"); - if (username == null) { - username = System.getenv("USER"); + log.info("End monitoring for task {}", pipelineTask); + // update processing state + pipelineTaskDataOperations().updateProcessingStep(pipelineTask, + ProcessingStep.WAITING_TO_STORE); + + // It may be the case that all the subtasks are processed, but that + // there are jobs still running, or (more likely) queued. We can address + // that by deleting them from PBS. + List jobIds = remoteJobIds( + incompleteRemoteJobs(jobsInformationByTask.get(pipelineTask))); + if (!CollectionUtils.isEmpty(jobIds)) { + queueCommandManager().deleteJobsByJobId(jobIds); } - return username; - } - public static void startLocalMonitoring(StateFile task) { - startMonitoring(task, AlgorithmType.LOCAL); - } + if (jobsInformationByTask.containsKey(pipelineTask)) { + updateRemoteJobs(pipelineTask); + } - public static void startRemoteMonitoring(StateFile task) { - startMonitoring(task, AlgorithmType.REMOTE); - } + taskMonitorByTask.remove(pipelineTask); - private static void startMonitoring(StateFile task, AlgorithmType algorithmType) { - AlgorithmMonitor instance = algorithmType.equals(AlgorithmType.REMOTE) - ? remoteMonitoringInstance - : localMonitoringInstance; - instance.startMonitoring(task); + // Figure out what needs to happen next, and do it. + determineDisposition(pipelineTask).performActions(this, pipelineTask); } - void startMonitoring(StateFile task) { - log.info("Starting monitoring for: " + task.invariantPart() + " on " + monitorVersion - + " algorithm monitor"); - state.put(task.invariantPart(), new StateFile(task)); - jobMonitor().addToMonitoring(task); + private void updateRemoteJobs(PipelineTask pipelineTask) { + pipelineTaskDataOperations().updateJobs(pipelineTask, true); } - private List stateFiles() { - - List stateDirListing = new LinkedList<>(); + /** + * Returns the collection of remote job IDs that correspond to a given collection of pipeline + * tasks, as a {@link Map}. + */ + public Map> jobIdsByTaskId(Collection pipelineTasks) { + Map> jobIdsByTaskId = new HashMap<>(); + if (jobsInformationByTask.size() == 0) { + return jobIdsByTaskId; + } + for (PipelineTask pipelineTask : pipelineTasks) { + List remoteJobsInformation = jobsInformationByTask + .get(pipelineTask); + if (remoteJobsInformation != null) { + jobIdsByTaskId.put(pipelineTask, + remoteJobIds(incompleteRemoteJobs(remoteJobsInformation))); + } + } + return jobIdsByTaskId; + } - // get the raw list, excluding directories - File stateDirFile = DirectoryProperties.stateFilesDir().toFile(); - List allFiles = new ArrayList<>(); - if (stateDirFile != null) { - File[] files = stateDirFile.listFiles(); - if (files != null) { - allFiles.addAll(Arrays.asList(files)); + private List incompleteRemoteJobs( + Collection remoteJobsInformation) { + List incompleteRemoteJobs = new ArrayList<>(); + if (CollectionUtils.isEmpty(remoteJobsInformation)) { + return incompleteRemoteJobs; + } + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + if (!Files.exists(Paths.get(remoteJobInformation.getLogFile()))) { + incompleteRemoteJobs.add(remoteJobInformation); } - stateDirListing = allFiles.stream() - .filter(s -> !s.isDirectory()) - .collect(Collectors.toList()); } + return incompleteRemoteJobs; + } - // throw away everything that doesn't have the correct pattern, and everything - // that is on the corrupted list - List filteredFiles = stateDirListing.stream() - .filter(s -> StateFile.STATE_FILE_NAME_PATTERN.matcher(s.getName()).matches()) - .filter(s -> !corruptedStateFileNames.contains(s.getName())) + private List remoteJobIds(Collection remoteJobsInformation) { + return remoteJobsInformation.stream() + .map(RemoteJobInformation::getJobId) .collect(Collectors.toList()); - return new LinkedList<>(filteredFiles); } - private void performStateFileChecks(StateFile oldState, StateFile remoteState) { - - String key = remoteState.invariantPart(); + private void checkForFinishedJobs() { - if (!oldState.equals(remoteState)) { - // state change - log.info("Updating state for: " + remoteState + " (was: " + oldState + ")"); - state.put(key, remoteState); - - long taskId = remoteState.getPipelineTaskId(); - - pipelineTaskOperations().updateSubtaskCounts(taskId, remoteState.getNumTotal(), - remoteState.getNumComplete(), remoteState.getNumFailed()); - - if (remoteState.isRunning()) { - // update processing state - pipelineTaskOperations().updateProcessingStep(taskId, ProcessingStep.EXECUTING); + for (PipelineTask pipelineTask : jobsInformationByTask.keySet()) { + if (isFinished(pipelineTask)) { + publishFinishedJobsMessage(pipelineTask); } + } + } - if (remoteState.isDone()) { - // update processing state - pipelineTaskOperations().updateProcessingStep(taskId, - ProcessingStep.WAITING_TO_STORE); - - // It may be the case that all the subtasks are processed, but that - // there are jobs still running, or (more likely) queued. We can address - // that by deleting them from PBS. - Set jobIds = jobMonitor().allIncompleteJobIds(remoteState); - if (jobIds != null && !jobIds.isEmpty()) { - jobMonitor().getQstatCommandManager().deleteJobsByJobId(jobIds); - } - - // Always send the task back to the worker - sendTaskToWorker(remoteState); - - log.info("Removing monitoring for: " + key); + private void publishFinishedJobsMessage(PipelineTask pipelineTask) { + allJobsFinishedMessage = new AllJobsFinishedMessage(pipelineTask); + ZiggyMessenger.publish(allJobsFinishedMessage, false); + } - state.remove(key); - jobMonitor().endMonitoring(remoteState); + private boolean isFinished(PipelineTask pipelineTask) { + List remoteJobsInformation = jobsInformationByTask.get(pipelineTask); + if (CollectionUtils.isEmpty(remoteJobsInformation)) { + return false; + } + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + if (!Files.exists(Paths.get(remoteJobInformation.getLogFile()))) { + return false; } - } else if (jobMonitor().isFinished(remoteState)) { - // Some job failures leave the state file untouched. If this happens, the QstatMonitor - // can determine that in fact the job is no longer running. In this case, set the state - // file to FAILED. Then in the next pass through this loop, standard handling for a - // failed job can be applied. - moveStateFileToCompleteState(remoteState); } + return true; } @Override @@ -310,118 +290,51 @@ public void run() { startLogMessageWritten = true; } try { - if (!state.isEmpty()) { - jobMonitor().update(); - - List stateDirListing = stateFiles(); - if (log.isDebugEnabled()) { - dumpRemoteState(stateDirListing); - } - performStateFileLoop(stateDirListing); - } + checkForFinishedJobs(); } catch (Exception e) { - // We don't want transient problems with the remote monitoring tool - // (which is third party software and not under our control) to bring - // down the monitor, so we catch all exceptions here including runtime - // ones in the hope and expectation that the next time we call the - // monitor the transient problem will have resolved itself. log.warn("Task monitor: exception has occurred", e); } } - /** - * Loops over all files in the stateDirListing and, if they are in the monitoring list, performs - * state file checks on the cached and file states. Any state file name that cannot be parsed - * into a new StateFile object is added to a registry of corrupted names and subsequently - * ignored. - * - * @param stateDirListing - */ - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - private void performStateFileLoop(List stateDirListing) { - for (File remoteFile : stateDirListing) { - String name = remoteFile.getName(); - try { - StateFile remoteState = new StateFile(name); - StateFile oldState = state.get(remoteState.invariantPart()); - if (oldState != null) { // ignore tasks we were not - // charged with - performStateFileChecks(oldState, remoteState); - } - } catch (Exception e) { - log.error("State file with name " + name - + " encountered exception and will be removed from monitoring", e); - corruptedStateFileNames.add(name); - } - } - } - - /** - * Resubmits a task to the worker and, optionally, to the NAS. This method is called both for - * complete tasks and failing tasks, because each needs to be looked at again by the worker. If - * the task has failing subtasks which should be resubmitted, the caller should specify - * true for restart. - * - * @param remoteState the state file - * @param restart if true, resubmit the task in the NAS - */ - private void sendTaskToWorker(StateFile remoteState) { - - PipelineTask pipelineTask = pipelineTaskOperations() - .pipelineTask(remoteState.getPipelineTaskId()); - - pipelineTaskOperations().updateJobs(pipelineTask, true); - - // Perform the actions necessary based on the task disposition - determineDisposition(remoteState, pipelineTask).performActions(this, pipelineTask); - } - /** * Handles a task for which processing has failed (either due to error or because the user * killed it). In this case, several actions need to be taken: the information about the cause * of the error has to be captured via qstat and logged locally; the pipeline task entry in the * database needs its TaskExecutionLog updated; the remote state file needs to be renamed to * indicate that the job errored. - * - * @param stateFile StateFile instance for deleted task. */ - private void handleFailedTask(StateFile stateFile) { + private void handleFailedTask(PipelineTask pipelineTask) { - // get the exit code and comment via qstat - String exitStatus = taskStatusValues(stateFile); - String exitComment = taskCommentValues(stateFile); - String exitState = stateFile.getState().toString().toLowerCase(); + // Get the exit code and comment. + String exitStatus = taskStatusValues(pipelineTask); + String exitComment = taskCommentValues(pipelineTask); - if (exitState.equals("deleted")) { - log.error("Task " + stateFile.getPipelineTaskId() + " has state file in " - + exitState.toUpperCase() + " state"); - } else { - log.error("Task " + stateFile.getPipelineTaskId() + " has failed"); - exitState = "failed"; - } + log.error("Task {} has failed", pipelineTask); if (exitStatus != null) { - log.error("Exit status from remote system for all jobs: " + exitStatus); + log.error("Exit status from remote system for all jobs is {}", exitStatus); } else { log.error("No exit status provided"); exitStatus = "not provided"; } if (exitComment != null) { - log.error("Exit comment from remote system: " + exitComment); + log.error("Exit comment from remote system is {}", exitComment); } else { log.error("No exit comment provided"); exitComment = "not provided"; } // issue an alert about the deletion - String message = algorithmType.equals(AlgorithmType.REMOTE) - ? "Task " + exitState + ", return codes = " + exitStatus + ", comments = " + exitComment - : "Task " + exitState; - alertService().generateAndBroadcastAlert("Algorithm Monitor", stateFile.getPipelineTaskId(), - AlertService.Severity.ERROR, message); + String message = pipelineTaskDataOperations() + .algorithmType(pipelineTask) == AlgorithmType.REMOTE + ? "Task failed, return codes = " + exitStatus + ", comments = " + exitComment + : "Task failed"; + alertService().generateAndBroadcastAlert("Algorithm Monitor", pipelineTask, Severity.ERROR, + message); } - private String taskStatusValues(StateFile stateFile) { - Map exitStatusByJobId = jobMonitor().exitStatus(stateFile); + private String taskStatusValues(PipelineTask pipelineTask) { + Map exitStatusByJobId = pbsLogParser() + .exitStatusByJobId(jobsInformationByTask.get(pipelineTask)); if (exitStatusByJobId.isEmpty()) { return null; } @@ -438,13 +351,14 @@ private String taskStatusValues(StateFile stateFile) { return sb.toString(); } - private String taskCommentValues(StateFile stateFile) { - Map exitComment = jobMonitor().exitComment(stateFile); - if (exitComment == null || exitComment.size() == 0) { + private String taskCommentValues(PipelineTask pipelineTask) { + Map exitCommentByJobId = pbsLogParser() + .exitCommentByJobId(jobsInformationByTask.get(pipelineTask)); + if (exitCommentByJobId == null || exitCommentByJobId.size() == 0) { return null; } StringBuilder sb = new StringBuilder(); - for (Map.Entry entry : exitComment.entrySet()) { + for (Map.Entry entry : exitCommentByJobId.entrySet()) { sb.append(entry.getKey()); sb.append("("); if (entry.getValue() != null) { @@ -456,62 +370,47 @@ private String taskCommentValues(StateFile stateFile) { return sb.toString(); } - /** - * Moves the state file for a remote job to the COMPLETE state. This is only done if the job - * ended on the remote system in a way that was not detected by the job itself but was later - * detected via qstat calls. - * - * @param stateFile State file for failed job. - */ - private void moveStateFileToCompleteState(StateFile stateFile) { - stateFile.setStateAndPersist(StateFile.State.COMPLETE); - } - - private Disposition determineDisposition(StateFile state, PipelineTask pipelineTask) { + private Disposition determineDisposition(PipelineTask pipelineTask) { // A task that was deliberately killed must be marked as failed regardless of // how many subtasks completed. - if (taskIsKilled(pipelineTask.getId())) { + if (taskIsKilled(pipelineTask)) { + log.debug("Task {} was halted", pipelineTask.getId()); + disposition = Disposition.FAIL; return Disposition.FAIL; } // The total number of bad subtasks includes both the ones that failed and the // ones that never ran / never finished. If there are few enough bad subtasks, // then we can persist results. - PipelineDefinitionNodeExecutionResources resources = pipelineTaskOperations() .executionResources(pipelineTask); - if (state.getNumTotal() - state.getNumComplete() <= resources.getMaxFailedSubtaskCount()) { + SubtaskCounts subtaskCounts = pipelineTaskDataOperations().subtaskCounts(pipelineTask); + log.debug("Number of subtasks for task {}: {}", pipelineTask.getId(), + subtaskCounts.getTotalSubtaskCount()); + log.debug("Number of completed subtasks for task {}: {}", pipelineTask.getId(), + subtaskCounts.getCompletedSubtaskCount()); + log.debug("Number of failed subtasks for task {}: {}", pipelineTask.getId(), + subtaskCounts.getFailedSubtaskCount()); + if (subtaskCounts.getTotalSubtaskCount() + - subtaskCounts.getCompletedSubtaskCount() <= resources.getMaxFailedSubtaskCount()) { + disposition = Disposition.PERSIST; return Disposition.PERSIST; } // If the task has bad subtasks but the number of automatic resubmits hasn't // been exhausted, then resubmit. - if (pipelineTask.getAutoResubmitCount() < resources.getMaxAutoResubmits()) { + if (pipelineTaskDataOperations().autoResubmitCount(pipelineTask) < resources + .getMaxAutoResubmits()) { + disposition = Disposition.RESUBMIT; return Disposition.RESUBMIT; } // If we've gotten this far, then the task has to be considered as failed: // it has too many bad subtasks and has exhausted its automatic retries. + disposition = Disposition.FAIL; return Disposition.FAIL; } - private void dumpRemoteState(List remoteState) { - log.debug("Remote state dir:"); - for (File file : remoteState) { - log.debug(file.toString()); - } - } - - /** - * Determines whether to continue the monitoring while-loop. For testing purposes, this can be - * replaced with a mocked version that performs a finite number of loops. - * - * @return true - */ - boolean continueMonitoring() { - return true; - } - /** * Obtains a new PipelineExecutor. Replace with mocked method for unit testing. * @@ -527,41 +426,63 @@ AlertService alertService() { } /** Replace with mocked method for unit testing. */ - boolean taskIsKilled(long taskId) { - return PipelineSupervisor.taskOnKilledTaskList(taskId); + boolean taskIsKilled(PipelineTask pipelineTask) { + return PipelineSupervisor.taskOnHaltedTaskList(pipelineTask); } - /** - * Returns the polling interval, in milliseconds. Replace with mocked method for unit testing. - */ - long pollingIntervalMillis() { - return algorithmType.equals(AlgorithmType.REMOTE) ? SSH_POLL_INTERVAL_MILLIS - : LOCAL_POLL_INTERVAL_MILLIS; + long finishedJobsPollingIntervalMillis() { + return FINISHED_JOBS_POLL_INTERVAL_MILLIS; } - /** Stops the thread pool and replaces it. For testing only. */ - static void resetThreadPool() { - if (threadPool != null) { - threadPool.shutdownNow(); - } - threadPool = new ScheduledThreadPoolExecutor(2); + long remotePollIntervalMillis() { + return REMOTE_POLL_INTERVAL_MILLIS; } - /** Gets the {@link StateFile} from the state {@link Map}. For testing only. */ - StateFile getStateFile(StateFile stateFile) { - return state.get(stateFile.invariantPart()); + long localPollIntervalMillis() { + return LOCAL_POLL_INTERVAL_MILLIS; } - private void initializeJobMonitor() { - jobMonitor = JobMonitor.newInstance(username(), algorithmType); - } - - /** Replace with mocked method for unit testing. */ - JobMonitor jobMonitor() { - return jobMonitor; + TaskMonitor taskMonitor(MonitorAlgorithmRequest monitorAlgorithmRequest) { + return new TaskMonitor(monitorAlgorithmRequest.getPipelineTask(), + monitorAlgorithmRequest.getTaskDir().toFile(), + CollectionUtils.isEmpty(monitorAlgorithmRequest.getRemoteJobsInformation()) + ? localPollIntervalMillis() + : remotePollIntervalMillis()); } PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + PbsLogParser pbsLogParser() { + return pbsLogParser; + } + + QueueCommandManager queueCommandManager() { + return queueCommandManager; + } + + // For testing only. + Map getTaskMonitorByTask() { + return taskMonitorByTask; + } + + // For testing only. + Map> getJobsInformationByTask() { + return jobsInformationByTask; + } + + // For testing only. + Disposition getDisposition() { + return disposition; + } + + // For testing only. + AllJobsFinishedMessage allJobsFinishedMessage() { + return allJobsFinishedMessage; + } } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java index c31e99b..b009064 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java @@ -12,7 +12,7 @@ /** * This class manages zero-length files whose names are used to represent the state of an executing - * algorithm. These files are stored in the subtask working directory. + * algorithm. These files are stored in the task and subtask working directories. * * @author Todd Klaus * @author PT @@ -22,7 +22,7 @@ public class AlgorithmStateFiles { private static final String HAS_OUTPUTS = "HAS_OUTPUTS"; - public enum SubtaskState { + public enum AlgorithmState { // State in which no AlgorithmStateFile is present. Rather than return an actual // null, when queried about the subtask state we can return SubtaskState.NULL . NULL { @@ -59,9 +59,9 @@ public void updateStateCounts(SubtaskStateCounts stateCounts) { private final File outputsFlag; public AlgorithmStateFiles(File workingDir) { - processingFlag = new File(workingDir, "." + SubtaskState.PROCESSING.toString()); - completeFlag = new File(workingDir, "." + SubtaskState.COMPLETE.toString()); - failedFlag = new File(workingDir, "." + SubtaskState.FAILED.toString()); + processingFlag = new File(workingDir, "." + AlgorithmState.PROCESSING.toString()); + completeFlag = new File(workingDir, "." + AlgorithmState.COMPLETE.toString()); + failedFlag = new File(workingDir, "." + AlgorithmState.FAILED.toString()); outputsFlag = new File(workingDir, "." + HAS_OUTPUTS); } @@ -81,7 +81,7 @@ public void clearState() { /** * Removes any "stale" state flags. A stale state flag is one from a previous processing attempt - * that will cause the pipeline to either miscount the task/ sub-task, or do the wrong thing + * that will cause the pipeline to either miscount the task/ subtask, or do the wrong thing * with it. COMPLETED states are never stale, because they finished and don't need to be * restarted. FAILED and PROCESSING flags are stale, because they indicate incomplete prior * execution but can prevent the current execution attempt from starting. @@ -91,7 +91,7 @@ public void clearState() { * the preceding run. */ public void clearStaleState() { - if (!currentSubtaskState().equals(SubtaskState.COMPLETE)) { + if (!currentAlgorithmState().equals(AlgorithmState.COMPLETE)) { outputsFlag.delete(); } processingFlag.delete(); @@ -99,7 +99,7 @@ public void clearStaleState() { } @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void updateCurrentState(SubtaskState newState) { + public void updateCurrentState(AlgorithmState newState) { clearState(); try { @@ -135,27 +135,27 @@ public void setOutputsFlag() { } } - public SubtaskState currentSubtaskState() { - SubtaskState current = SubtaskState.NULL; + public AlgorithmState currentAlgorithmState() { + AlgorithmState current = AlgorithmState.NULL; if (processingFlag.exists()) { - current = SubtaskState.PROCESSING; + current = AlgorithmState.PROCESSING; } if (completeFlag.exists()) { - if (current != SubtaskState.NULL) { + if (current != AlgorithmState.NULL) { log.warn("Duplicate algorithm state files found!"); return null; } - current = SubtaskState.COMPLETE; + current = AlgorithmState.COMPLETE; } if (failedFlag.exists()) { - if (current != SubtaskState.NULL) { + if (current != AlgorithmState.NULL) { log.warn("Duplicate algorithm state files found!"); return null; } - current = SubtaskState.FAILED; + current = AlgorithmState.FAILED; } - log.debug("current state: " + current); + log.debug("current={}", current); return current; } @@ -164,20 +164,20 @@ public SubtaskState currentSubtaskState() { * * @return */ - public boolean subtaskStateExists() { + public boolean stateExists() { return completeFlag.exists() || processingFlag.exists() || failedFlag.exists(); } public boolean isProcessing() { - return currentSubtaskState() == SubtaskState.PROCESSING; + return currentAlgorithmState() == AlgorithmState.PROCESSING; } public boolean isComplete() { - return currentSubtaskState() == SubtaskState.COMPLETE; + return currentAlgorithmState() == AlgorithmState.COMPLETE; } public boolean isFailed() { - return currentSubtaskState() == SubtaskState.FAILED; + return currentAlgorithmState() == AlgorithmState.FAILED; } /** diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmType.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmType.java new file mode 100644 index 0000000..1c42931 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmType.java @@ -0,0 +1,9 @@ +package gov.nasa.ziggy.module; + +public enum AlgorithmType { + /** Local execution only. */ + LOCAL, + + /** Execution occurs on another host or system. */ + REMOTE +} diff --git a/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java b/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java index ab2af5f..77d61a6 100644 --- a/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java +++ b/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java @@ -34,34 +34,33 @@ package gov.nasa.ziggy.module; +import java.io.BufferedReader; import java.io.File; +import java.io.FileInputStream; import java.io.IOException; -import java.nio.file.Paths; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; import java.util.HashSet; import java.util.Set; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledThreadPoolExecutor; -import java.util.concurrent.Semaphore; import java.util.concurrent.ThreadFactory; -import java.util.concurrent.TimeUnit; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.google.common.util.concurrent.ThreadFactoryBuilder; -import gov.nasa.ziggy.module.StateFile.State; -import gov.nasa.ziggy.module.remote.TimestampFile; +import gov.nasa.ziggy.module.AlgorithmStateFiles.AlgorithmState; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.logging.TaskLog; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.HostNameUtils; -import gov.nasa.ziggy.util.TimeFormatter; -import gov.nasa.ziggy.util.io.LockManager; +import gov.nasa.ziggy.util.io.ZiggyFileUtils; /** * Acts as a controller for a single-node remote job and associated subtasks running on the node. @@ -78,24 +77,18 @@ * @author Todd Klaus * @author PT */ -public class ComputeNodeMaster implements Runnable { +public class ComputeNodeMaster { private static final Logger log = LoggerFactory.getLogger(ComputeNodeMaster.class); - private static final long SLEEP_INTERVAL_MILLIS = 10000; - private final String workingDir; private int coresPerNode; - private final StateFile stateFile; private final File taskDir; - private final File stateFileLockFile; private String nodeName = ""; - private TaskMonitor monitor; private SubtaskServer subtaskServer; - private Semaphore subtaskMasterSemaphore; - private CountDownLatch monitoringLatch = new CountDownLatch(1); + private CountDownLatch subtaskMasterCountdownLatch; private ExecutorService threadPool; private Set subtaskMasters = new HashSet<>(); @@ -106,209 +99,107 @@ public ComputeNodeMaster(String workingDir) { this.workingDir = workingDir; log.info("RemoteTaskMaster START"); - log.info(" workingDir = " + workingDir); + log.info(" workingDir = {}", workingDir); + + // Tell anyone who cares that this task is no longer queued. + new AlgorithmStateFiles(new File(workingDir)).updateCurrentState(AlgorithmState.PROCESSING); nodeName = HostNameUtils.shortHostName(); - stateFile = StateFile.of(Paths.get(workingDir)).newStateFileFromDiskFile(); taskDir = new File(workingDir); - stateFileLockFile = stateFile.lockFile(); } /** - * Initializes the {@link ComputeNodeMaster}. Specifically, it locates the file that carries the - * node name of the node with the {@link SubtaskServer} and starts new threads for the - * {@link SubtaskMaster} instances. For the node that is going to host the {@link SubtaskServer} - * instance it also starts the server, updates the {@link StateFile}, creates symlinks, and - * creates task-start timestamps. - *

- * The {@link #initialize()} method returns a boolean that indicates whether monitoring is - * required. This returns true if there are unprocessed subtasks and false if all subtasks are - * actually processed. + * Initializes the {@link ComputeNodeMaster}. A number of timestamp files are created if needed + * in the task directory; a {@link SubtaskServer} instance is created in for the node; a + * collection of {@link SubtaskMaster} instances are started. */ - public boolean initialize() { + public void initialize() { - log.info("Starting ComputeNodeMaster ({})", - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + log.info("Starting ComputeNodeMaster ({}, {})", BuildInfo.ziggyVersion(), + BuildInfo.pipelineVersion()); ZiggyConfiguration.logJvmProperties(); - // It's possible that this node isn't starting until all of the subtasks are - // complete! In that case, it should just exit without doing anything else. - monitor = new TaskMonitor(stateFile, taskDir); - monitor.updateState(); - if (monitor.allSubtasksProcessed()) { - StateFile updatedStateFile = new StateFile(stateFile); - updatedStateFile.setState(StateFile.State.COMPLETE); - StateFile.updateStateFile(stateFile, updatedStateFile); - log.info("All subtasks processed, ComputeNodeMaster exiting"); - return false; - } - - coresPerNode = stateFile.getActiveCoresPerNode(); + coresPerNode = activeCoresFromFile(); - updateStateFile(); subtaskServer().start(); createTimestamps(); - log.info("Starting " + coresPerNode + " subtask masters"); + log.info("Starting {} subtask masters", coresPerNode); startSubtaskMasters(); - return true; } - /** - * Moves the state file for this task from QUEUED to PROCESSING. - * - * @throws IOException if unable to release write lock on state file. - */ - private void updateStateFile() { + private void createTimestamps() { + long arriveTime = System.currentTimeMillis(); - // NB: If there are multiple jobs associated with a single task, this update only - // needs to be performed if this job is the first to start - boolean stateFileLockObtained = getWriteLockWithoutBlocking(stateFileLockFile); - try { - StateFile previousStateFile = new StateFile(stateFile); - if (stateFileLockObtained - && previousStateFile.getState().equals(StateFile.State.QUEUED)) { - stateFile.setState(StateFile.State.PROCESSING); - log.info("Updating state: " + previousStateFile + " -> " + stateFile); - - if (!StateFile.updateStateFile(previousStateFile, stateFile)) { - log.error("Failed to update state file: " + previousStateFile); - } - } else { - log.info("State file already moved to PROCESSING state, not modifying state file"); - stateFile.setState(StateFile.State.PROCESSING); - } - } finally { - if (stateFileLockObtained) { - releaseWriteLock(stateFileLockFile); - } - } + TimestampFile.create(taskDir, TimestampFile.Event.ARRIVE_COMPUTE_NODES, arriveTime); + TimestampFile.create(taskDir, TimestampFile.Event.START); } - private void createTimestamps() { - long arriveTime = stateFile.getPfeArrivalTimeMillis() != StateFile.INVALID_VALUE - ? stateFile.getPfeArrivalTimeMillis() - : System.currentTimeMillis(); - long submitTime = stateFile.getPbsSubmitTimeMillis(); - - TimestampFile.create(taskDir, TimestampFile.Event.ARRIVE_PFE, arriveTime); - TimestampFile.create(taskDir, TimestampFile.Event.QUEUED_PBS, submitTime); - TimestampFile.create(taskDir, TimestampFile.Event.PBS_JOB_START); + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private int activeCoresFromFile() { + try (BufferedReader reader = new BufferedReader(new InputStreamReader( + new FileInputStream( + taskDir.toPath().resolve(AlgorithmExecutor.ACTIVE_CORES_FILE_NAME).toFile()), + ZiggyFileUtils.ZIGGY_CHARSET))) { + String fileText = reader.readLine(); + return Integer.parseInt(fileText); + } catch (IOException e) { + throw new UncheckedIOException(e); + } } /** * Starts the {@link SubtaskMaster} instances in the threads of a thread pool, one thread per - * active cores on this node. A {@link Semaphore} is used to track the number of + * active cores on this node. A {@link CountDownLatch} is used to track the number of * {@link SubtaskMaster} instances currently running. */ @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) private void startSubtaskMasters() { - int timeoutSecs = (int) TimeFormatter - .timeStringHhMmSsToTimeInSeconds(stateFile.getRequestedWallTime()); - subtaskMasterSemaphore = new Semaphore(coresPerNode); + int timeoutSecs = wallTimeFromFile(); + subtaskMasterCountdownLatch = new CountDownLatch(coresPerNode); threadPool = subtaskMasterThreadPool(); ThreadFactory threadFactory = new ThreadFactoryBuilder().setNameFormat("SubtaskMaster[%d]") .build(); + String executableName = ZiggyConfiguration.getInstance() + .getString(PropertyName.ZIGGY_ALGORITHM_NAME.property()); for (int i = 0; i < coresPerNode; i++) { - try { - subtaskMasterSemaphore.acquire(); - } catch (InterruptedException e) { - // This can never occur. The number of permits is equal to the number of threads, - // thus there is no need to wait for a permit to become available. - throw new AssertionError(e); - } - SubtaskMaster subtaskMaster = new SubtaskMaster(i, nodeName, subtaskMasterSemaphore, - stateFile.getExecutableName(), workingDir, timeoutSecs); + SubtaskMaster subtaskMaster = new SubtaskMaster(i, nodeName, + subtaskMasterCountdownLatch, executableName, workingDir, timeoutSecs); subtaskMasters.add(subtaskMaster); threadPool.submit(subtaskMaster, threadFactory); } } - /** - * Performs periodic checks of subtask processing status. This is accomplished by using a - * {@link ScheduledThreadPoolExecutor} to check processing at the desired intervals. Execution - * of the {@link TaskMonitor} thread will block until the monitoring checks determine that - * processing is completed, at which time the thread resumes execution. - *

- * The specific conditions under which the monitor will resume execution of the current thread - * are as follows: - *

    - *
  1. All of the {@link SubtaskMaster} threads have completed. - *
  2. All of the subtasks are either completed or failed. - *
- */ - @AcceptableCatchBlock(rationale = Rationale.SYSTEM_EXIT) - public void monitor() { - - log.info("Waiting for subtasks to complete"); - - ScheduledThreadPoolExecutor monitoringThreadPool = new ScheduledThreadPoolExecutor(1); - monitoringThreadPool.scheduleAtFixedRate(this, 0L, SLEEP_INTERVAL_MILLIS, - TimeUnit.MILLISECONDS); - try { - monitoringLatch.await(); - } catch (InterruptedException e) { - // If the ComputeNodeMaster main thread is interrupted, it means that the entire - // ComputeNodeMaster is shutting down. We can simply allow it to shut down and don't - // need to do anything further. + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private int wallTimeFromFile() { + try (BufferedReader reader = new BufferedReader(new InputStreamReader( + new FileInputStream( + taskDir.toPath().resolve(AlgorithmExecutor.WALL_TIME_FILE_NAME).toFile()), + ZiggyFileUtils.ZIGGY_CHARSET))) { + String fileText = reader.readLine(); + return Integer.parseInt(fileText); + } catch (IOException e) { + throw new UncheckedIOException(e); } - monitoringThreadPool.shutdownNow(); } - @Override - public void run() { - - if (monitoringLatch.getCount() == 0) { - return; - } - - // If the subtask server has failed, then we - // don't need to do any finalization, just exit and start the process of all subtask - // master threads stopping. - if (!subtaskServer().isListenerRunning()) { - log.error("ComputeNodeMaster: error exit"); - endMonitoring(); - return; - } - - // Do state checks and updates - boolean allSubtasksProcessed = monitor.allSubtasksProcessed(); - monitor.updateState(); - - // If all the subtasks are either completed or failed, exit monitoring - // immediately - if (allSubtasksProcessed) { - log.info("All subtasks complete"); - endMonitoring(); - return; - } - - // If all RemoteSubtaskMasters are done we can exit monitoring - if (allPermitsAvailable()) { - endMonitoring(); + public void awaitSubtaskMastersComplete() { + try { + subtaskMasterCountdownLatch.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); } } - /** - * Ends monitoring. This is accomplished by decrementing the {@link CountDownLatch} that the - * main thread is waiting for. - */ - private synchronized void endMonitoring() { - monitoringLatch.countDown(); - } - /** * If monitoring ended with successful completion of the job, create a timestamp for the - * completion time in the task directory and mark the task's {@link StateFile} as done. + * completion time in the task directory. */ public void finish() { - TimestampFile.create(taskDir, TimestampFile.Event.PBS_JOB_FINISH); - if (monitor.allSubtasksProcessed()) { - monitor.markStateFileDone(); - } + TimestampFile.create(taskDir, TimestampFile.Event.FINISH); log.info("ComputeNodeMaster: Done"); } @@ -323,40 +214,10 @@ public void cleanup() { subtaskServer().shutdown(); } - // The following getter methods are intended for testing purposes only. They do not expose any - // of the ComputeNodeMaster's private objects to callers. In some cases it is necessary for - // the methods to be public, as they are used by tests in other packages. - public long getCountDownLatchCount() { - return monitoringLatch.getCount(); - } - - public int getSemaphorePermits() { - if (subtaskMasterSemaphore == null) { - return -1; - } - return subtaskMasterSemaphore.availablePermits(); - } - int subtaskMastersCount() { return subtaskMasters.size(); } - State getStateFileState() { - return stateFile.getState(); - } - - int getStateFileNumComplete() { - return stateFile.getNumComplete(); - } - - int getStateFileNumFailed() { - return stateFile.getNumFailed(); - } - - int getStateFileNumTotal() { - return stateFile.getNumTotal(); - } - /** * Restores the {@link TaskConfigurationHandler} from disk. Package scope so it can be replaced * with a mocked instance. @@ -368,31 +229,6 @@ TaskConfiguration getTaskConfiguration() { return inputsHandler; } - /** - * Attempts to obtain the lock file for the task's state file, but does not block if it cannot - * obtain it. Broken out as a separate method to support testing. - * - * @return true if lock obtained, false otherwise. - */ - boolean getWriteLockWithoutBlocking(File lockFile) { - return LockManager.getWriteLockWithoutBlocking(lockFile); - } - - /** - * Releases the write lock on a file. Broken out as a separate method to support testing. - */ - void releaseWriteLock(File lockFile) { - LockManager.releaseWriteLock(lockFile); - } - - /** - * Determines whether all permits are available for the {@link Semaphore} that works with the - * {@link SubtaskMaster} instances. Broken out as a separate method to support testing. - */ - boolean allPermitsAvailable() { - return subtaskMasterSemaphore.availablePermits() == coresPerNode; - } - /** * Returns a new instance of {@link SubtaskServer}. Broken out as a separate method to support * testing. @@ -424,10 +260,9 @@ public static void main(String[] args) { ComputeNodeMaster computeNodeMaster = null; // Startup: constructor and initialization - boolean monitoringRequired = false; try { computeNodeMaster = new ComputeNodeMaster(workingDir); - monitoringRequired = computeNodeMaster.initialize(); + computeNodeMaster.initialize(); } catch (Exception e) { // Any exception that occurs in the constructor or initialization is @@ -439,11 +274,8 @@ public static void main(String[] args) { System.exit(1); } - // Monitoring: wait for subtasks to finish, subtask masters to finish, or exceptions - // to be thrown - if (monitoringRequired) { - computeNodeMaster.monitor(); - } + // Wait for the subtask masters to finish. + computeNodeMaster.awaitSubtaskMastersComplete(); // Wrap-up: finalize and clean up computeNodeMaster.finish(); diff --git a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java index 629ace9..5a75c83 100644 --- a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java +++ b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java @@ -147,10 +147,10 @@ public void copyDatastoreFilesToTaskDirectory(TaskConfiguration taskConfiguratio public SubtaskInformation subtaskInformation(PipelineDefinitionNode pipelineDefinitionNode) { if (pipelineDefinitionNode.getSingleSubtask()) { return new SubtaskInformation(getPipelineTask().getModuleName(), - getPipelineTask().uowTaskInstance().briefState(), 1); + getPipelineTask().getUnitOfWork().briefState(), 1); } return new SubtaskInformation(getPipelineTask().getModuleName(), - getPipelineTask().uowTaskInstance().briefState(), + getPipelineTask().getUnitOfWork().briefState(), datastoreFileManager().subtaskCount(pipelineDefinitionNode)); } diff --git a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java index afe7e98..9646bf4 100644 --- a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java +++ b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java @@ -4,7 +4,7 @@ import java.util.Collection; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java b/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java index 43e5d2a..f5974fe 100644 --- a/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java @@ -31,13 +31,13 @@ import gov.nasa.ziggy.metrics.IntervalMetric; import gov.nasa.ziggy.metrics.Metric; import gov.nasa.ziggy.metrics.ValueMetric; -import gov.nasa.ziggy.module.remote.TimestampFile; import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; @@ -90,7 +90,7 @@ public boolean processTask() { @Override public List restartModes() { return List.of(RunMode.RESTART_FROM_BEGINNING, RunMode.RESUBMIT, - RunMode.RESUME_CURRENT_STEP, RunMode.RESUME_MONITORING); + RunMode.RESUME_CURRENT_STEP); } protected File getTaskDir() { @@ -138,11 +138,11 @@ public void marshalingTaskAction() { // Set the next step, whatever it might be. incrementProcessingStep(); - // If there are sub-task inputs, then we can go on to the next step. + // If there are subtask inputs, then we can go on to the next step. successful = true; } else { - // If there are no sub-task inputs, we should stop processing. + // If there are no subtask inputs, we should stop processing. successful = false; checkHaltRequest(ProcessingStep.MARSHALING); } @@ -158,7 +158,7 @@ public void copyFilesToTaskDirectory(TaskConfiguration taskConfiguration, File taskWorkingDirectory) { pipelineInputs().copyDatastoreFilesToTaskDirectory(taskConfiguration, taskWorkingDirectory.toPath()); - pipelineTaskOperations().updateSubtaskCounts(pipelineTask().getId(), + pipelineTaskDataOperations().updateSubtaskCounts(pipelineTask, taskConfiguration.getSubtaskCount(), 0, 0); } @@ -211,7 +211,7 @@ public void executingTaskAction() { checkHaltRequest(ProcessingStep.EXECUTING); doneLooping = true; processingSuccessful = false; - log.info("Resubmitting {} algorithm to remote system", ProcessingStep.EXECUTING); + log.info("Resubmitting {} algorithm to remote system...done", ProcessingStep.EXECUTING); } /** @@ -240,19 +240,19 @@ public void storingTaskAction() { // add metrics for "RemoteWorker", "PleiadesQueue", "Matlab", // "PendingReceive" - long remoteWorkerTime = timestampFileElapsedTimeMillis(TimestampFile.Event.ARRIVE_PFE, - TimestampFile.Event.QUEUED_PBS); - long pleiadesQueueTime = timestampFileElapsedTimeMillis(TimestampFile.Event.QUEUED_PBS, - TimestampFile.Event.PBS_JOB_START); - long pleiadesWallTime = timestampFileElapsedTimeMillis( - TimestampFile.Event.PBS_JOB_START, TimestampFile.Event.PBS_JOB_FINISH); + long remoteWorkerTime = timestampFileElapsedTimeMillis( + TimestampFile.Event.ARRIVE_COMPUTE_NODES, TimestampFile.Event.QUEUED); + long pleiadesQueueTime = timestampFileElapsedTimeMillis(TimestampFile.Event.QUEUED, + TimestampFile.Event.START); + long pleiadesWallTime = timestampFileElapsedTimeMillis(TimestampFile.Event.START, + TimestampFile.Event.FINISH); long pendingReceiveTime = startTransferTime - - timestampFileTimestamp(TimestampFile.Event.PBS_JOB_FINISH); + - timestampFileTimestamp(TimestampFile.Event.FINISH); - log.info("remoteWorkerTime = " + remoteWorkerTime); - log.info("pleiadesQueueTime = " + pleiadesQueueTime); - log.info("pleiadesWallTime = " + pleiadesWallTime); - log.info("pendingReceiveTime = " + pendingReceiveTime); + log.info("remoteWorkerTime = {}", remoteWorkerTime); + log.info("pleiadesQueueTime = {}", pleiadesQueueTime); + log.info("pleiadesWallTime = {}", pleiadesWallTime); + log.info("pendingReceiveTime = {}", pendingReceiveTime); valueMetricAddValue(REMOTE_WORKER_WAIT_METRIC, remoteWorkerTime); valueMetricAddValue(PLEIADES_QUEUE_METRIC, pleiadesQueueTime); @@ -262,9 +262,9 @@ public void storingTaskAction() { ProcessingFailureSummary failureSummary = processingFailureSummary(); boolean abandonPersisting = false; if (!failureSummary.isAllTasksSucceeded() && !failureSummary.isAllTasksFailed()) { - log.info("Sub-task failures occurred. List of sub-task failures follows:"); - for (String failedSubTask : failureSummary.getFailedSubTaskDirs()) { - log.info(" " + failedSubTask); + log.info("Subtask failures occurred. List of subtask failures follows:"); + for (String failedSubtask : failureSummary.getFailedSubtaskDirs()) { + log.info(" {}", failedSubtask); } ImmutableConfiguration config = ZiggyConfiguration.getInstance(); boolean allowPartialTasks = config @@ -272,11 +272,11 @@ public void storingTaskAction() { abandonPersisting = !allowPartialTasks; } if (failureSummary.isAllTasksFailed()) { - log.info("All sub-tasks failed in processing, abandoning storage of results"); + log.info("All subtasks failed in processing, abandoning storage of results"); abandonPersisting = true; } if (abandonPersisting) { - throw new PipelineException("Unable to persist due to sub-task failures"); + throw new PipelineException("Unable to persist due to subtask failures"); } IntervalMetric.measure(STORE_OUTPUTS_METRIC, () -> { @@ -321,7 +321,7 @@ void persistResultsAndUpdateConsumers() { log.warn("Input file {} produced no output", inputFile.toString()); } AlertService.getInstance() - .generateAndBroadcastAlert("Algorithm", taskId(), AlertService.Severity.WARNING, + .generateAndBroadcastAlert("Algorithm", pipelineTask(), Severity.WARNING, inputFiles.getFilesWithoutOutputs() + " input files produced no output, see log for details"); } @@ -370,7 +370,7 @@ void processingMainLoop() { @Override protected void restartFromBeginning() { - pipelineTaskOperations().updateProcessingStep(taskId(), processingSteps().get(0)); + pipelineTaskDataOperations().updateProcessingStep(pipelineTask, processingSteps().get(0)); processingMainLoop(); } @@ -381,15 +381,10 @@ protected void resumeCurrentStep() { @Override protected void resubmit() { - pipelineTaskOperations().updateProcessingStep(taskId(), ProcessingStep.SUBMITTING); + pipelineTaskDataOperations().updateProcessingStep(pipelineTask, ProcessingStep.SUBMITTING); processingMainLoop(); } - @Override - protected void resumeMonitoring() { - algorithmManager().getExecutor().resumeMonitoring(); - } - @Override protected void runStandard() { processingMainLoop(); @@ -403,19 +398,19 @@ public void updateMetrics(PipelineTask pipelineTask, Map threadM throw new PipelineException("processTask called with incorrect pipeline task"); } - List summaryMetrics = pipelineTaskOperations() - .summaryMetrics(pipelineTask); + List pipelineTaskMetrics = pipelineTaskDataOperations() + .pipelineTaskMetrics(pipelineTask); log.debug("Thread Metrics:"); for (String threadMetricName : threadMetrics.keySet()) { - log.debug("TM: " + threadMetricName + ": " - + threadMetrics.get(threadMetricName).getLogString()); + log.debug("TM: {}: {}", threadMetricName, + threadMetrics.get(threadMetricName).getLogString()); } // cross-reference existing summary metrics by category - Map summaryMetricsByCategory = new HashMap<>(); - for (PipelineTaskMetrics summaryMetric : summaryMetrics) { - summaryMetricsByCategory.put(summaryMetric.getCategory(), summaryMetric); + Map pipelineTaskMetricByCategory = new HashMap<>(); + for (PipelineTaskMetric pipelineTaskMetric : pipelineTaskMetrics) { + pipelineTaskMetricByCategory.put(pipelineTaskMetric.getCategory(), pipelineTaskMetric); } String[] categories; @@ -444,16 +439,15 @@ public void updateMetrics(PipelineTask pipelineTask, Map threadM ValueMetric iMetric = (ValueMetric) metric; totalTime = iMetric.getSum(); } else { - log.info("Module did not provide metric with name = " + metricName); + log.info("Module did not provide metric with name = {}", metricName); } - log.info("TaskID={}, category={}, time(ms)={}", pipelineTask.getId(), category, - totalTime); + log.info("TaskID={}, category={}, time(ms)={}", pipelineTask, category, totalTime); - PipelineTaskMetrics m = summaryMetricsByCategory.get(category); + PipelineTaskMetric m = pipelineTaskMetricByCategory.get(category); if (m == null) { - m = new PipelineTaskMetrics(category, totalTime, unit); - summaryMetrics.add(m); + m = new PipelineTaskMetric(category, totalTime, unit); + pipelineTaskMetrics.add(m); } // don't overwrite the existing value if no value was recorded for @@ -463,8 +457,8 @@ public void updateMetrics(PipelineTask pipelineTask, Map threadM m.setValue(totalTime); } } - pipelineTask.setSummaryMetrics(summaryMetrics); - pipelineTaskOperations().merge(pipelineTask); + + pipelineTaskDataOperations().updatePipelineTaskMetrics(pipelineTask, pipelineTaskMetrics); } /** @@ -491,7 +485,7 @@ long timestampFileElapsedTimeMillis(TimestampFile.Event startEvent, } long timestampFileTimestamp(TimestampFile.Event event) { - return TimestampFile.timestamp(getTaskDir(), event); + return TimestampFile.timestamp(getTaskDir(), event, false); } ValueMetric valueMetricAddValue(String name, long value) { diff --git a/src/main/java/gov/nasa/ziggy/module/JobMonitor.java b/src/main/java/gov/nasa/ziggy/module/JobMonitor.java deleted file mode 100644 index e509bee..0000000 --- a/src/main/java/gov/nasa/ziggy/module/JobMonitor.java +++ /dev/null @@ -1,83 +0,0 @@ -package gov.nasa.ziggy.module; - -import java.util.Collections; -import java.util.Map; -import java.util.Set; - -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; -import gov.nasa.ziggy.module.remote.QstatMonitor; -import gov.nasa.ziggy.module.remote.QueueCommandManager; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.util.HostNameUtils; - -/** - * Interface for classes that monitor remote jobs. This allows a dummy implementation to be supplied - * in cases in which there are calls to a remote job monitor but no remote jobs to be monitored (see - * {@link AlgorithmMonitor} for more information). - * - * @author PT - */ -public interface JobMonitor { - - /** - * Returns a new instance of {@link JobMonitor} that is correct for its use-case based on - * arguments. In particular, for remote tasks an instance of {@link QstatMonitor} will be - * returned, while for local tasks a dummy instance, with no actual functionality, will be - * returned. - */ - static JobMonitor newInstance(String username, AlgorithmType algorithmType) { - if (algorithmType.equals(AlgorithmType.REMOTE)) { - return new QstatMonitor(username, HostNameUtils.shortHostName()); - } - return new JobMonitor() { - }; - } - - default void addToMonitoring(StateFile stateFile) { - } - - default void addToMonitoring(PipelineTask pipelineTask) { - } - - default void endMonitoring(StateFile stateFile) { - } - - default void update() { - } - - default Set allIncompleteJobIds(PipelineTask pipelineTask) { - return Collections.emptySet(); - } - - default Set allIncompleteJobIds(StateFile stateFile) { - return Collections.emptySet(); - } - - default boolean isFinished(StateFile stateFile) { - return false; - } - - default Map exitStatus(StateFile stateFile) { - return Collections.emptyMap(); - } - - default Map exitComment(StateFile stateFile) { - return Collections.emptyMap(); - } - - default String getOwner() { - return ""; - } - - default String getServerName() { - return ""; - } - - default QueueCommandManager getQstatCommandManager() { - return null; - } - - default Set getJobsInMonitor() { - return Collections.emptySet(); - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java b/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java index 6f10d52..d65f3a4 100644 --- a/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java @@ -12,6 +12,7 @@ import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.logging.TaskLog; import gov.nasa.ziggy.services.messages.MonitorAlgorithmRequest; @@ -30,6 +31,8 @@ public class LocalAlgorithmExecutor extends AlgorithmExecutor { private static final Logger log = LoggerFactory.getLogger(LocalAlgorithmExecutor.class); + private PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations = new PipelineModuleDefinitionOperations(); + public LocalAlgorithmExecutor(PipelineTask pipelineTask) { super(pipelineTask); } @@ -51,20 +54,18 @@ protected Path taskDataDir() { } @Override - protected void addToMonitor(StateFile stateFile) { - ZiggyMessenger.publish(new MonitorAlgorithmRequest(stateFile, algorithmType())); + protected void addToMonitor() { + ZiggyMessenger.publish(new MonitorAlgorithmRequest(pipelineTask, workingDir())); } @Override - protected void submitForExecution(StateFile stateFile) { + protected void submitForExecution() { - stateFile.setPbsSubmitTimeMillis(System.currentTimeMillis()); - stateFile.persist(); - addToMonitor(stateFile); + addToMonitor(); CommandLine cmdLine = algorithmCommandLine(); - pipelineTaskOperations().setLocalExecution(pipelineTask.getId()); + pipelineTaskDataOperations().updateAlgorithmType(pipelineTask, AlgorithmType.LOCAL); // Start the external process -- note that it will cause execution to block until // the algorithm has completed or failed. @@ -79,11 +80,9 @@ protected void submitForExecution(StateFile stateFile) { // as having failed if (exitCode != 0) { if (!ZiggyShutdownHook.shutdownInProgress()) { - throw new PipelineException( - "Local processing of task " + pipelineTask.getId() + " failed"); + throw new PipelineException("Local processing of task " + pipelineTask + " failed"); } - log.error( - "Task " + pipelineTask.getId() + " processing incomplete due to worker shutdown"); + log.error("Task {} processing incomplete due to worker shutdown", pipelineTask); } } @@ -114,4 +113,21 @@ protected CommandLine algorithmCommandLine() { public AlgorithmType algorithmType() { return AlgorithmType.LOCAL; } + + @Override + protected String activeCores() { + return "1"; + } + + @Override + protected String wallTime() { + return Integer.toString(pipelineModuleDefinitionOperations() + .pipelineModuleExecutionResources(pipelineModuleDefinitionOperations() + .pipelineModuleDefinition(pipelineTask.getModuleName())) + .getExeTimeoutSeconds()); + } + + PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations() { + return pipelineModuleDefinitionOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineCategories.java b/src/main/java/gov/nasa/ziggy/module/PipelineCategories.java index 1ec4ef1..574b591 100644 --- a/src/main/java/gov/nasa/ziggy/module/PipelineCategories.java +++ b/src/main/java/gov/nasa/ziggy/module/PipelineCategories.java @@ -1,6 +1,6 @@ package gov.nasa.ziggy.module; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; /** * Categories used by the pipeline in various contexts. diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java b/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java index 756a947..2a86a61 100644 --- a/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java @@ -30,14 +30,14 @@ public abstract class PipelineInputsOutputsUtils implements Persistable { private static final String SERIALIZED_OUTPUTS_TYPE_FILE = ".output-types.ser"; /** - * Returns the task directory. Assumes that the working directory is the sub-task directory. + * Returns the task directory. Assumes that the working directory is the subtask directory. */ public static Path taskDir() { return DirectoryProperties.workingDir().getParent(); } /** - * Returns the module executable name. Assumes that the working directory is the sub-task + * Returns the module executable name. Assumes that the working directory is the subtask * directory. */ public static String moduleName() { @@ -45,15 +45,11 @@ public static String moduleName() { } public static String moduleName(Path taskDir) { - String taskDirString = taskDir.getFileName().toString(); - PipelineTask.TaskBaseNameMatcher m = new PipelineTask.TaskBaseNameMatcher(taskDirString); - return m.moduleName(); + return new PipelineTask.TaskBaseNameMatcher(taskDir.getFileName().toString()).moduleName(); } public static long taskId(Path taskDir) { - String taskDirString = taskDir.getFileName().toString(); - PipelineTask.TaskBaseNameMatcher m = new PipelineTask.TaskBaseNameMatcher(taskDirString); - return m.taskId(); + return new PipelineTask.TaskBaseNameMatcher(taskDir.getFileName().toString()).taskId(); } /** diff --git a/src/main/java/gov/nasa/ziggy/module/ProcessingFailureSummary.java b/src/main/java/gov/nasa/ziggy/module/ProcessingFailureSummary.java index bf17eee..6daeb45 100644 --- a/src/main/java/gov/nasa/ziggy/module/ProcessingFailureSummary.java +++ b/src/main/java/gov/nasa/ziggy/module/ProcessingFailureSummary.java @@ -7,41 +7,41 @@ import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; /** - * Provides summary information on failed sub-tasks. + * Provides summary information on failed subtasks. * * @author PT */ public class ProcessingFailureSummary { - private List failedSubTaskDirs = new ArrayList<>(); + private List failedSubtaskDirs = new ArrayList<>(); private boolean allTasksFailed = false; public ProcessingFailureSummary(String moduleName, File taskDirectory) { SubtaskDirectoryIterator taskDirectoryIterator = new SubtaskDirectoryIterator( taskDirectory); - int numSubTasks = taskDirectoryIterator.numSubTasks(); + int numSubtasks = taskDirectoryIterator.numSubtasks(); - // loop over sub-task directories and look for stack trace files + // loop over subtask directories and look for stack trace files while (taskDirectoryIterator.hasNext()) { - File subTaskDir = taskDirectoryIterator.next().getSubtaskDir(); - if (ModuleInterfaceUtils.errorFile(subTaskDir, moduleName).exists()) { - failedSubTaskDirs.add(subTaskDir.getName()); + File subtaskDir = taskDirectoryIterator.next().getSubtaskDir(); + if (ModuleInterfaceUtils.errorFile(subtaskDir, moduleName).exists()) { + failedSubtaskDirs.add(subtaskDir.getName()); } } - allTasksFailed = failedSubTaskDirs.size() == numSubTasks; + allTasksFailed = failedSubtaskDirs.size() == numSubtasks; } public boolean isAllTasksFailed() { return allTasksFailed; } - public List getFailedSubTaskDirs() { - return failedSubTaskDirs; + public List getFailedSubtaskDirs() { + return failedSubtaskDirs; } public boolean isAllTasksSucceeded() { - return failedSubTaskDirs.size() == 0; + return failedSubtaskDirs.size() == 0; } } diff --git a/src/main/java/gov/nasa/ziggy/module/StateFile.java b/src/main/java/gov/nasa/ziggy/module/StateFile.java deleted file mode 100644 index 308459d..0000000 --- a/src/main/java/gov/nasa/ziggy/module/StateFile.java +++ /dev/null @@ -1,888 +0,0 @@ -package gov.nasa.ziggy.module; - -import java.io.File; -import java.io.FileFilter; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.OutputStreamWriter; -import java.io.Serializable; -import java.io.UncheckedIOException; -import java.io.Writer; -import java.nio.file.Path; -import java.util.ArrayList; -import java.util.Collections; -import java.util.Date; -import java.util.Iterator; -import java.util.List; -import java.util.Objects; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -import org.apache.commons.configuration2.PropertiesConfiguration; -import org.apache.commons.configuration2.builder.fluent.Configurations; -import org.apache.commons.configuration2.ex.ConfigurationException; -import org.apache.commons.io.FileUtils; -import org.apache.commons.io.filefilter.WildcardFileFilter; -import org.apache.commons.lang3.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.Iso8601Formatter; -import gov.nasa.ziggy.util.io.LockManager; -import gov.nasa.ziggy.util.io.ZiggyFileUtils; - -/** - * This class models a file whose name contains the state of a pipeline task executing on a remote - * cluster. - *

- * The file also contains additional properties of the remote job. - *

- * Each state file represents a single unit of work from the perspective of the pipeline module. - * - * 

- * Filename Format:
- *
- * ziggy-PIID.PTID.EXENAME.STATE_TOTAL-COMPLETE-FAILED
- *
- * PIID: Pipeline Instance ID
- * PTID: Pipeline Task ID
- * EXENAME: Name of the MATLAB executable
- * STATE: enum(SUBMITTED,PROCESSING,ERRORS_RUNNING,FAILED,COMPLETE)
- * TOTAL-COMPLETE-FAILED): Number of jobs in each category
- *
- * - * The file contains the properties that reflect the properties in this class, including: - *
- *
timeoutSecs
- *
Timeout for the MATLAB process.
- *
gigsPerCore
- *
Required memory per core used. Used to calculate coresPerNode based on architecture.
- *
tasksPerCore
- *
Number of tasks to allocate to each available core.
- *
remoteNodeArchitecture
- *
Architecture to use.
- *
remoteGroup
- *
Group name used for the qsub command on the remote node.
- *
queueName
- *
Queue name used for the qsub command on the remote node.
- *
reRunnable
- *
Whether this task is re-runnable.
- *
localBinToMatEnabled
- *
If true, don't generate .mat files on the remote node.
- *
requestedWallTime
- *
Requested wall time for the PBS qsub command.
- *
symlinksEnabled
- *
Determines whether symlinks are created between sub-task directories and files in the - * top-level task directory. This should be enabled for modules that store files common to all - * sub-tasks in the top-level task directory.
- *
pbsSubmitTimeMillis
- *
The time in milliseconds that the job was submitted to the Pleiades scheduler, the Portable - * Batch System (PBS).
- *
pfeArrivalTimeMillis
- *
This seems to be the same as pbsSubmitTimeMillis in most cases.
- *
- * - * @author Bill Wohler - * @author Todd Klaus - */ -public class StateFile implements Comparable, Serializable { - private static final long serialVersionUID = 20230511L; - - private static final Logger log = LoggerFactory.getLogger(StateFile.class); - - public static final Pattern TASK_DIR_PATTERN = Pattern.compile("([0-9]+)-([0-9]+)-(\\S+)"); - public static final int TASK_DIR_INSTANCE_ID_GROUP_NUMBER = 1; - public static final int TASK_DIR_TASK_ID_GROUP_NUMBER = 2; - public static final int TASK_DIR_MODULE_NAME_GROUP_NUMBER = 3; - - public static final String PREFIX_BARE = "ziggy"; - public static final String PREFIX = PREFIX_BARE + "."; - public static final String PREFIX_WITH_BACKSLASHES = PREFIX_BARE + "\\."; - - public static final String DEFAULT_REMOTE_NODE_ARCHITECTURE = "none"; - public static final String DEFAULT_WALL_TIME = "24:00:00"; - public static final String INVALID_STRING = "none"; - public static final int INVALID_VALUE = -1; - - public static final String LOCK_FILE_NAME = ".state-file.lock"; - - private static final String REMOTE_NODE_ARCHITECTURE_PROP_NAME = "remoteNodeArchitecture"; - private static final String MIN_CORES_PER_NODE_PROP_NAME = "minCoresPerNode"; - private static final String MIN_GIGS_PER_NODE_PROP_NAME = "minGigsPerNode"; - private static final String REMOTE_GROUP_PROP_NAME = "remoteGroup"; - private static final String QUEUE_NAME_PROP_NAME = "queueName"; - private static final String REQUESTED_WALLTIME_PROP_NAME = "requestedWallTime"; - private static final String REQUESTED_NODE_COUNT_PROP_NAME = "requestedNodeCount"; - private static final String ACTIVE_CORES_PER_NODE_PROP_NAME = "activeCoresPerNode"; - private static final String GIGS_PER_SUBTASK_PROP_NAME = "gigsPerSubtask"; - private static final String EXECUTABLE_NAME_PROP_NAME = "executableName"; - - private static final String PBS_SUBMIT_PROP_NAME = "pbsSubmitTimeMillis"; - private static final String PFE_ARRIVAL_PROP_NAME = "pfeArrivalTimeMillis"; - - public enum State { - /** Task has been initialized. */ - INITIALIZED, - - /** - * Task has been submitted by the worker, but not yet picked up by the remote cluster. - */ - SUBMITTED, - - /** Task is waiting for compute nodes to become available. */ - QUEUED, - - /** Task is running on the compute nodes. */ - PROCESSING, - - /** - * Task has finished running on the compute nodes, either with or without subtask errors. - */ - COMPLETE, - - /** The final state for this task has been acknowledged by the worker. */ - CLOSED; - - } - - private static String statesPatternElement; - - // Concatenate all of the states into a single String for use in the state file - // name pattern. - static { - StringBuilder sb = new StringBuilder(); - for (State state : State.values()) { - sb.append(state.toString()); - sb.append("|"); - } - sb.setLength(sb.length() - 1); - statesPatternElement = sb.toString(); - } - - // Pattern and regex for a state file name - private static final String STATE_FILE_NAME_REGEX = PREFIX_WITH_BACKSLASHES - + "([0-9]+)\\.([0-9]+)\\.(\\S+)\\." + "(" + statesPatternElement + ")" - + "_([0-9]+)-([0-9]+)-([0-9]+)"; - public static final Pattern STATE_FILE_NAME_PATTERN = Pattern.compile(STATE_FILE_NAME_REGEX); - private static final int STATE_FILE_NAME_INSTANCE_ID_GROUP_NUMBER = 1; - private static final int STATE_FILE_NAME_TASK_ID_GROUP_NUMBER = 2; - private static final int STATE_FILE_NAME_MODULE_GROUP_NUMBER = 3; - private static final int STATE_FILE_NAME_STATE_GROUP_NUMBER = 4; - private static final int STATE_FILE_NAME_TOTAL_SUBTASKS_GROUP_NUMBER = 5; - private static final int STATE_FILE_NAME_COMPLETE_SUBTASKS_GROUP_NUMBER = 6; - private static final int STATE_FILE_NAME_FAILED_SUBTASKS_GROUP_NUMBER = 7; - - // This is a slightly more informative explanation of the file name format, - // used in log files so the user knows exactly what was expected. - private static final String FORMAT = PREFIX + "PIID.PTID.MODNAME.STATE_TOTAL-COMPLETE-FAILED"; - - // Fields in the file name. - private long pipelineInstanceId = 0; - private long pipelineTaskId = 0; - private String moduleName; - private State state = State.INITIALIZED; - private int numTotal = 0; - private int numComplete = 0; - private int numFailed = 0; - - /** Contains all properties from the file. */ - private transient PropertiesConfiguration props = new PropertiesConfiguration(); - - public StateFile() { - } - - /** - * Constructs a StateFile containing only the invariant part. - * - * @param moduleName the name of the pipeline module for the task - * @param pipelineInstanceId the pipeline instance ID - * @param pipelineTaskId the pipeline task ID - */ - public StateFile(String moduleName, long pipelineInstanceId, long pipelineTaskId) { - this.moduleName = moduleName; - this.pipelineInstanceId = pipelineInstanceId; - this.pipelineTaskId = pipelineTaskId; - } - - /** - * Constructs a {@link StateFile} instance from a task directory. The task directory name is - * parsed to obtain the module name, instance ID, and task ID components. - */ - public static StateFile of(Path taskDir) { - String taskDirName = taskDir.getFileName().toString(); - Matcher matcher = TASK_DIR_PATTERN.matcher(taskDirName); - if (!matcher.matches()) { - throw new IllegalArgumentException( - "Task dir name " + taskDirName + " does not match convention for task dir names"); - } - return new StateFile(matcher.group(TASK_DIR_MODULE_NAME_GROUP_NUMBER), - Long.parseLong(matcher.group(TASK_DIR_INSTANCE_ID_GROUP_NUMBER)), - Long.parseLong(matcher.group(TASK_DIR_TASK_ID_GROUP_NUMBER))); - } - - /** - * Constructs a StateFile from an existing name. - */ - public StateFile(String name) { - parse(name); - } - - /** - * Parses a string of the form: PREFIX + MODNAME.PIID.PTID.STATE_TOTAL-COMPLETE-FAILED) - */ - private void parse(String name) { - Matcher matcher = STATE_FILE_NAME_PATTERN.matcher(name); - if (!matcher.matches()) { - throw new IllegalArgumentException(name + " does not match expected format: " + FORMAT); - } - - pipelineInstanceId = Long - .parseLong(matcher.group(STATE_FILE_NAME_INSTANCE_ID_GROUP_NUMBER)); - pipelineTaskId = Long.parseLong(matcher.group(STATE_FILE_NAME_TASK_ID_GROUP_NUMBER)); - moduleName = matcher.group(STATE_FILE_NAME_MODULE_GROUP_NUMBER); - state = State.valueOf(matcher.group(STATE_FILE_NAME_STATE_GROUP_NUMBER)); - numTotal = Integer.parseInt(matcher.group(STATE_FILE_NAME_TOTAL_SUBTASKS_GROUP_NUMBER)); - numComplete = Integer - .parseInt(matcher.group(STATE_FILE_NAME_COMPLETE_SUBTASKS_GROUP_NUMBER)); - numFailed = Integer.parseInt(matcher.group(STATE_FILE_NAME_FAILED_SUBTASKS_GROUP_NUMBER)); - } - - /** - * Creates a shallow copy of the given object. - */ - public StateFile(StateFile other) { - - // members - - moduleName = other.getModuleName(); - pipelineInstanceId = other.getPipelineInstanceId(); - pipelineTaskId = other.getPipelineTaskId(); - state = other.getState(); - numTotal = other.getNumTotal(); - numComplete = other.getNumComplete(); - numFailed = other.getNumFailed(); - - // properties of the PropertiesConfiguration - - List propertyNames = other.getSortedPropertyNames(); - for (String propertyName : propertyNames) { - props.setProperty(propertyName, other.props.getProperty(propertyName)); - } - } - - /** - * Gets the property names of the StateFile's PropertyConfigurations object, sorted - * alphabetically - * - * @return a List of property names defined for this object. - */ - private List getSortedPropertyNames() { - Iterator propertyNamesIterator = props.getKeys(); - List propertyNames = new ArrayList<>(); - while (propertyNamesIterator.hasNext()) { - propertyNames.add(propertyNamesIterator.next()); - } - Collections.sort(propertyNames); - return propertyNames; - } - - /** - * Creates a StateFile from the given parameters. - */ - public static StateFile generateStateFile(PipelineTask pipelineTask, - PbsParameters pbsParameters, int numSubtasks) { - - StateFile state = new StateFile.Builder().moduleName(pipelineTask.getModuleName()) - .executableName(pipelineTask.getExecutableName()) - .pipelineInstanceId(pipelineTask.getPipelineInstanceId()) - .pipelineTaskId(pipelineTask.getId()) - .numTotal(numSubtasks) - .numComplete(0) - .numFailed(0) - .state(StateFile.State.QUEUED) - .build(); - - if (pbsParameters != null) { - state.setActiveCoresPerNode(pbsParameters.getActiveCoresPerNode()); - state.setRemoteNodeArchitecture(pbsParameters.getArchitecture().getNodeName()); - state.setMinCoresPerNode(pbsParameters.getMinCoresPerNode()); - state.setMinGigsPerNode(pbsParameters.getMinGigsPerNode()); - state.setRemoteGroup(pbsParameters.getRemoteGroup()); - state.setQueueName(pbsParameters.getQueueName()); - state.setRequestedWallTime(pbsParameters.getRequestedWallTime()); - state.setRequestedNodeCount(pbsParameters.getRequestedNodeCount()); - state.setGigsPerSubtask(pbsParameters.getGigsPerSubtask()); - } else { - state.setActiveCoresPerNode(1); - state.setRemoteNodeArchitecture(""); - state.setRemoteGroup(""); - state.setQueueName(""); - } - - return state; - } - - /** - * Persists this {@link StateFile} to the state file directory. - */ - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public File persist() { - File directory = DirectoryProperties.stateFilesDir().toFile(); - File file = new File(directory, name()); - try (Writer fw = new OutputStreamWriter(new FileOutputStream(file), - ZiggyFileUtils.ZIGGY_CHARSET)) { - props.write(fw); - - // Also, move any old state files that are for the same instance and - // task as this one. - moveOldStateFiles(directory); - } catch (ConfigurationException e) { - // This can never occur. The construction of the props field is guaranteed - // to be correct. - throw new AssertionError(e); - } catch (IOException e) { - throw new UncheckedIOException("Unable to write to file " + file.toString(), e); - } - return file; - } - - private void moveOldStateFiles(File stateFileDir) { - - String stateFileName = name(); - FileFilter fileFilter = new WildcardFileFilter(invariantPart() + "*"); - File[] matches = stateFileDir.listFiles(fileFilter); - - if (matches == null || matches.length == 0) { - throw new PipelineException( - "State file \"" + stateFileName + "\" does not exist or there was an I/O error."); - } - - String iso8601Date = Iso8601Formatter.dateTimeLocalFormatter().format(new Date()); - int fileCounter = 0; - - // For all the matched files that are NOT the current state file, rename - // the old ones to a new name that removes the "ziggy" at the beginning - // (replacing it with "old"), and appends a datestamp and index #. - - for (File match : matches) { - if (!match.getName().equals(stateFileName)) { - String nameSansPrefix = match.getName().substring(PREFIX.length()); - String newName = "old." + nameSansPrefix + "." + iso8601Date + "." - + Integer.toString(fileCounter); - File newFile = new File(stateFileDir, newName); - match.renameTo(newFile); - log.warn("File " + match.getName() + " in directory " + stateFileDir - + " renamed to " + newFile.getName()); - } - } - } - - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static boolean updateStateFile(StateFile oldStateFile, StateFile newStateFile) { - - File stateFileDir = DirectoryProperties.stateFilesDir().toFile(); - if (oldStateFile.equals(newStateFile)) { - log.debug("Old state file " + oldStateFile.name() + " is the same as new state file " - + newStateFile.name() + ", not changing"); - } - // Update the state file. - log.info("Updating state: " + oldStateFile + " -> " + newStateFile); - File oldFile = new File(stateFileDir, oldStateFile.name()); - File newFile = new File(stateFileDir, newStateFile.name()); - - log.debug(" renaming file: " + oldFile + " -> " + newFile); - - try { - FileUtils.moveFile(oldFile, newFile); - } catch (IOException e) { - throw new UncheckedIOException("Unable to move file " + oldFile.toString(), e); - } - - return true; - } - - /** - * Returns a {@link List} of {@link StateFile} instances from the state files directory that are - * in the {@link State#PROCESSING} state. - */ - public static List processingStateFilesFromDisk() { - File directory = DirectoryProperties.stateFilesDir().toFile(); - - String[] filenames = directory.list((dir, name) -> { - Matcher matcher = STATE_FILE_NAME_PATTERN.matcher(name); - return matcher.matches() && matcher.group(STATE_FILE_NAME_STATE_GROUP_NUMBER) - .equals(State.PROCESSING.toString()); - }); - List stateFiles = new ArrayList<>(); - for (String filename : filenames) { - stateFiles.add(new StateFile(filename)); - } - return stateFiles; - } - - /** - * Constructs a new {@link StateFile} from an existing file. - * - * @return a {@link StateFile} object derived from the specified file on disk - */ - public StateFile newStateFileFromDiskFile() { - return newStateFileFromDiskFile(false); - } - - /** - * Constructs a new {@link StateFile} from an existing file. - * - * @param silent when true suppresses the logging message from matching the disk file. - * @return a StateFile object derived from the specified file on disk - */ - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - public StateFile newStateFileFromDiskFile(boolean silent) { - - File stateFilePathToUse = StateFile.getDiskFileFromInvariantNamePart(this); - - if (!silent) { - log.info("Matched statefile: " + stateFilePathToUse); - } - StateFile stateFile = new StateFile(stateFilePathToUse.getName()); - try { - PropertiesConfiguration props = new Configurations().properties(stateFilePathToUse); - if (props.isEmpty()) { - throw new PipelineException("State file contains no properties!"); - } - stateFile.props = props; - } catch (ConfigurationException e) { - // This can never occur. By construction, the props field is guaranteee - // to be constructed correctly. - throw new AssertionError(e); - } - - return stateFile; - } - - /** - * Searches a directory for a state file where the name's invariant part matches that provided - * by the caller. - * - * @param oldStateFile existing {@link StateFile} instance - * @return the file in the directory of the stateFilePath that has a name for which the - * invariant part matches the invariant part of the stateFilePath name (i.e., - * ziggy.pa.6.23.QUEUED.10.0.0 will match ziggy.pa.6.23.* on disk) - * @exception IllegalArgumentException if the stateFilePath's parent is not a directory - * @exception IllegalStateException if there isn't only one StateFile that matches the invariant - * part of stateFilePath - */ - private static File getDiskFileFromInvariantNamePart(StateFile oldStateFile) { - - // Sadly, this is probably the easiest way to do this. - String invariantPart = oldStateFile.invariantPart(); - File directory = DirectoryProperties.stateFilesDir().toFile(); - if (!directory.exists() || !directory.isDirectory()) { - throw new IllegalArgumentException( - "Specified directory does not exist or is not a directory: " + directory); - } - - FileFilter fileFilter = new WildcardFileFilter(invariantPart + "*"); - File[] matches = directory.listFiles(fileFilter); - if (matches == null) { - throw new IllegalStateException( - "No state file matching " + invariantPart + "* in directory " + directory); - } - if (matches.length > 1) { - throw new IllegalStateException("More than one state file matches " + invariantPart - + "* in directory " + directory); - } - - return matches[0]; - } - - /** - * Builds the name of the state file based on the elements. - */ - public String name() { - return invariantPart() + "." + state + "_" + numTotal + "-" + numComplete + "-" + numFailed; - } - - /** - * Returns the invariant part of the state file name. This includes the static PREFIX and the - * pipeline instance and task ids, plus the module name. - */ - public String invariantPart() { - return PREFIX + pipelineInstanceId + "." + pipelineTaskId + "." + moduleName; - } - - public static String invariantPart(PipelineTask task) { - return PREFIX + task.getPipelineInstanceId() + "." + task.getId() + "." - + task.getModuleName(); - } - - public String taskBaseName() { - return PipelineTask.taskBaseName(pipelineInstanceId, pipelineTaskId, moduleName); - } - - /** - * Returns the name of the task dir represented by this StateFile. - */ - public String taskDirName() { - return PipelineTask.taskBaseName(getPipelineInstanceId(), getPipelineTaskId(), - getModuleName()); - } - - public void setStateAndPersist(State state) { - LockManager.getWriteLockOrBlock(lockFile()); - try { - StateFile oldState = newStateFileFromDiskFile(true); - StateFile newState = new StateFile(oldState); - newState.setState(state); - StateFile.updateStateFile(oldState, newState); - } finally { - LockManager.releaseWriteLock(lockFile()); - } - } - - public File lockFile() { - return DirectoryProperties.taskDataDir() - .resolve(taskDirName()) - .resolve(LOCK_FILE_NAME) - .toFile(); - } - - public boolean isDone() { - return state == State.COMPLETE || state == State.CLOSED; - } - - public boolean isRunning() { - return state == State.PROCESSING; - } - - public boolean isQueued() { - return state == State.QUEUED; - } - - public boolean isStarted() { - return isRunning() || isDone(); - } - - @Override - public int compareTo(StateFile o) { - return name().compareTo(o.name()); - } - - @Override - public String toString() { - return name(); - } - - public long getPipelineInstanceId() { - return pipelineInstanceId; - } - - public void setPipelineInstanceId(long pipelineInstanceId) { - this.pipelineInstanceId = pipelineInstanceId; - } - - public long getPipelineTaskId() { - return pipelineTaskId; - } - - public void setPipelineTaskId(long pipelineTaskId) { - this.pipelineTaskId = pipelineTaskId; - } - - public String getModuleName() { - return moduleName; - } - - public void setModuleName(String moduleName) { - this.moduleName = moduleName; - } - - public State getState() { - return state; - } - - public void setState(State state) { - this.state = state; - } - - public int getNumTotal() { - return numTotal; - } - - public void setNumTotal(int numTotal) { - this.numTotal = numTotal; - } - - public int getNumComplete() { - return numComplete; - } - - public void setNumComplete(int numComplete) { - this.numComplete = numComplete; - } - - public int getNumFailed() { - return numFailed; - } - - public void setNumFailed(int numFailed) { - this.numFailed = numFailed; - } - - public String getExecutableName() { - return props.getProperty(EXECUTABLE_NAME_PROP_NAME) != null - && !StringUtils.isBlank(props.getString(EXECUTABLE_NAME_PROP_NAME)) - ? props.getString(EXECUTABLE_NAME_PROP_NAME) - : INVALID_STRING; - } - - public void setExecutableName(String executableName) { - props.setProperty(EXECUTABLE_NAME_PROP_NAME, executableName); - } - - /** - * Returns the value of the {@value #REMOTE_NODE_ARCHITECTURE_PROP_NAME} property, or - * {@link #DEFAULT_REMOTE_NODE_ARCHITECTURE} if not present or set. - */ - public String getRemoteNodeArchitecture() { - return props.getProperty(REMOTE_NODE_ARCHITECTURE_PROP_NAME) != null - && !props.getString(REMOTE_NODE_ARCHITECTURE_PROP_NAME).isBlank() - ? props.getString(REMOTE_NODE_ARCHITECTURE_PROP_NAME) - : DEFAULT_REMOTE_NODE_ARCHITECTURE; - } - - public void setRemoteNodeArchitecture(String remoteNodeArchitecture) { - props.setProperty(REMOTE_NODE_ARCHITECTURE_PROP_NAME, remoteNodeArchitecture); - } - - /** - * Returns the value of the {@value #MIN_CORES_PER_NODE_PROP_NAME} property, or 0 if not present - * or set. - */ - public int getMinCoresPerNode() { - return props.getProperty(MIN_CORES_PER_NODE_PROP_NAME) != null - ? props.getInt(MIN_CORES_PER_NODE_PROP_NAME) - : INVALID_VALUE; - } - - public void setMinCoresPerNode(int minCoresPerNode) { - props.setProperty(MIN_CORES_PER_NODE_PROP_NAME, minCoresPerNode); - } - - /** - * Returns the value of the {@value #MIN_GIGS_PER_NODE_PROP_NAME} property, or 0 if not present - * or set. - */ - public double getMinGigsPerNode() { - return props.getProperty(MIN_GIGS_PER_NODE_PROP_NAME) != null - ? props.getDouble(MIN_GIGS_PER_NODE_PROP_NAME) - : INVALID_VALUE; - } - - public void setMinGigsPerNode(double minGigsPerNode) { - props.setProperty(MIN_GIGS_PER_NODE_PROP_NAME, minGigsPerNode); - } - - /** - * Returns the value of the {@value #REQUESTED_WALLTIME_PROP_NAME} property, or - * {@link #DEFAULT_WALL_TIME} if not present or set. - */ - public String getRequestedWallTime() { - return props.getProperty(REQUESTED_WALLTIME_PROP_NAME) != null - ? props.getString(REQUESTED_WALLTIME_PROP_NAME) - : DEFAULT_WALL_TIME; - } - - public void setRequestedWallTime(String requestedWallTime) { - props.setProperty(REQUESTED_WALLTIME_PROP_NAME, requestedWallTime); - } - - /** - * Returns the value of the {@value #REMOTE_GROUP_PROP_NAME} property, or - * {@link #INVALID_STRING} if not present or set. - */ - public String getRemoteGroup() { - return props.getProperty(REMOTE_GROUP_PROP_NAME) != null - ? props.getString(REMOTE_GROUP_PROP_NAME) - : INVALID_STRING; - } - - public void setRemoteGroup(String remoteGroup) { - props.setProperty(REMOTE_GROUP_PROP_NAME, remoteGroup); - } - - /** - * Returns the value of the {@value #QUEUE_NAME_PROP_NAME} property, or {@link #INVALID_STRING} - * if not present or set. - */ - public String getQueueName() { - return props.getProperty(QUEUE_NAME_PROP_NAME) != null - ? props.getString(QUEUE_NAME_PROP_NAME) - : INVALID_STRING; - } - - public void setQueueName(String queueName) { - props.setProperty(QUEUE_NAME_PROP_NAME, queueName); - } - - public int getActiveCoresPerNode() { - return props.getProperty(ACTIVE_CORES_PER_NODE_PROP_NAME) != null - ? props.getInt(ACTIVE_CORES_PER_NODE_PROP_NAME) - : INVALID_VALUE; - } - - public void setActiveCoresPerNode(int activeCoresPerNode) { - props.setProperty(ACTIVE_CORES_PER_NODE_PROP_NAME, activeCoresPerNode); - } - - public int getRequestedNodeCount() { - return props.getProperty(REQUESTED_NODE_COUNT_PROP_NAME) != null - ? props.getInt(REQUESTED_NODE_COUNT_PROP_NAME) - : INVALID_VALUE; - } - - public void setRequestedNodeCount(int requestedNodeCount) { - props.setProperty(REQUESTED_NODE_COUNT_PROP_NAME, requestedNodeCount); - } - - public double getGigsPerSubtask() { - return props.getProperty(GIGS_PER_SUBTASK_PROP_NAME) != null - ? props.getDouble(GIGS_PER_SUBTASK_PROP_NAME) - : INVALID_VALUE; - } - - public void setGigsPerSubtask(double gigsPerSubtask) { - props.setProperty(GIGS_PER_SUBTASK_PROP_NAME, gigsPerSubtask); - } - - /** - * Returns the value of the {@value #PBS_SUBMIT_PROP_NAME} property, or {@link #INVALID_VALUE} - * if not present or set. - */ - public long getPbsSubmitTimeMillis() { - return props.getProperty(PBS_SUBMIT_PROP_NAME) != null ? props.getLong(PBS_SUBMIT_PROP_NAME) - : INVALID_VALUE; - } - - public void setPbsSubmitTimeMillis(long pbsSubmitTimeMillis) { - props.setProperty(PBS_SUBMIT_PROP_NAME, pbsSubmitTimeMillis); - } - - /** - * Returns the value of the {@value #PFE_ARRIVAL_PROP_NAME} property, or {@link #INVALID_VALUE} - * if not present or set. - */ - public long getPfeArrivalTimeMillis() { - return props.getProperty(PFE_ARRIVAL_PROP_NAME) != null - ? props.getLong(PFE_ARRIVAL_PROP_NAME) - : INVALID_VALUE; - } - - public void setPfeArrivalTimeMillis(long pfeArrivalTimeMillis) { - props.setProperty(PFE_ARRIVAL_PROP_NAME, pfeArrivalTimeMillis); - } - - @Override - public int hashCode() { - return Objects.hash(moduleName, numComplete, numFailed, numTotal, pipelineInstanceId, - pipelineTaskId, state); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - StateFile other = (StateFile) obj; - if (moduleName == null) { - if (other.moduleName == null) { - return false; - } - } else if (!moduleName.equals(other.moduleName)) { - return false; - } - if (numComplete != other.numComplete || numFailed != other.numFailed - || numTotal != other.numTotal || pipelineInstanceId != other.pipelineInstanceId) { - return false; - } - if (pipelineTaskId != other.pipelineTaskId || state != other.state) { - return false; - } - return true; - } - - /** - * Used to construct a {@link StateFile} object. To use this class, a {@link StateFile} object - * is created and then non-null fields are set using the available builder methods. Finally, a - * {@link StateFile} object is created using the {@code build} method. For example: - * - * 
-     * StateFile stateFile = new StateFile.Builder().foo(fozar).bar(bazar).build();
-     *
- * - * This pattern is based upon - * - * Josh Bloch's JavaOne 2006 talk, Effective Java Reloaded, TS-1512. - * - * @author PT - */ - public static class Builder { - - private StateFile stateFile = new StateFile(); - - public Builder() { - } - - public Builder moduleName(String moduleName) { - stateFile.setModuleName(moduleName); - return this; - } - - public Builder executableName(String executableName) { - stateFile.setExecutableName(executableName); - return this; - } - - public Builder pipelineInstanceId(long pipelineInstanceId) { - stateFile.setPipelineInstanceId(pipelineInstanceId); - return this; - } - - public Builder pipelineTaskId(long pipelineTaskId) { - stateFile.setPipelineTaskId(pipelineTaskId); - return this; - } - - public Builder state(State state) { - stateFile.setState(state); - return this; - } - - public Builder numTotal(int numTotal) { - stateFile.setNumTotal(numTotal); - return this; - } - - public Builder numComplete(int numComplete) { - stateFile.setNumComplete(numComplete); - return this; - } - - public Builder numFailed(int numFailed) { - stateFile.setNumFailed(numFailed); - return this; - } - - public StateFile build() { - return new StateFile(stateFile); - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java b/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java index c863515..32e5a8f 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java @@ -35,34 +35,34 @@ public SubtaskAllocator(TaskConfiguration taskConfiguration) { } } - public boolean markSubtaskComplete(int subTaskIndex) { - boolean found = markSubtaskNeedsNoFurtherProcessing(subTaskIndex); - subtaskCompleted[subTaskIndex] = true; + public boolean markSubtaskComplete(int subtaskIndex) { + boolean found = markSubtaskNeedsNoFurtherProcessing(subtaskIndex); + subtaskCompleted[subtaskIndex] = true; return found; } - public boolean markSubtaskLocked(int subTaskIndex) { - return markSubtaskNeedsNoFurtherProcessing(subTaskIndex); + public boolean markSubtaskLocked(int subtaskIndex) { + return markSubtaskNeedsNoFurtherProcessing(subtaskIndex); } - private boolean markSubtaskNeedsNoFurtherProcessing(int subTaskIndex) { + private boolean markSubtaskNeedsNoFurtherProcessing(int subtaskIndex) { boolean found = false; for (int i = 0; i < currentPoolProcessing.size(); i++) { - if (currentPoolProcessing.get(i) == subTaskIndex) { + if (currentPoolProcessing.get(i) == subtaskIndex) { found = true; currentPoolProcessing.remove(i); - log.debug("removing subtaskIndex: " + subTaskIndex); + log.debug("Removing subtaskIndex {}", subtaskIndex); } } if (!found) { - log.warn("failed to remove subtaskIndex: " + subTaskIndex); + log.warn("Failed to remove subtaskIndex {}", subtaskIndex); return false; } return true; } /** - * Return the next sub-task available for processing + * Return the next subtask available for processing * * @return */ diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskClient.java b/src/main/java/gov/nasa/ziggy/module/SubtaskClient.java index 0d42a36..34b89a1 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskClient.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskClient.java @@ -29,6 +29,9 @@ public class SubtaskClient { private static final Logger log = LoggerFactory.getLogger(SubtaskClient.class); + // How long should the client wait after a TRY_AGAIN response? + public static final long TRY_AGAIN_WAIT_TIME_MILLIS = 2000L; + // The SubtaskClient only handles one request / response at a time, hence the // ArrayBlockingQueue needs only one entry. private final ArrayBlockingQueue responseQueue = new ArrayBlockingQueue<>(1); @@ -38,22 +41,23 @@ public SubtaskClient() { } /** - * Client method to report that a sub-task has completed. + * Client method to report that a subtask has completed. */ - public Response reportSubtaskComplete(int subTaskIndex) { - return request(RequestType.REPORT_DONE, subTaskIndex); + public Response reportSubtaskComplete(int subtaskIndex) { + return request(RequestType.REPORT_DONE, subtaskIndex); } /** - * Client method to report that a sub-task is locked by another compute node. + * Client method to report that a subtask is locked by another compute node. */ - public Response reportSubtaskLocked(int subTaskIndex) { - return request(RequestType.REPORT_LOCKED, subTaskIndex); + public Response reportSubtaskLocked(int subtaskIndex) { + return request(RequestType.REPORT_LOCKED, subtaskIndex); } /** * Get the next subtask for processing. */ + @AcceptableCatchBlock(rationale = Rationale.CLEANUP_BEFORE_EXIT) public Response nextSubtask() { Response response = null; @@ -62,8 +66,13 @@ public Response nextSubtask() { if (response == null || response.status != ResponseType.TRY_AGAIN) { break; } + try { + Thread.sleep(tryAgainWaitTimeMillis()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } } - log.debug("getNextSubTask: Got a response: " + response); + log.debug("getNextSubtask: Got a response: {}", response); return response; } @@ -94,7 +103,7 @@ private Response request(RequestType command) { */ private Response request(RequestType command, int subtaskIndex) { - log.debug("Sending request " + command + " with subtaskIndex " + subtaskIndex); + log.debug("Sending request {} with subtaskIndex {}", command, subtaskIndex); send(command, subtaskIndex); return receive(); } @@ -103,9 +112,9 @@ private Response request(RequestType command, int subtaskIndex) { * Sends a request to the {@link SubtaskServer}. This is acomplished by creating a new instance * of {@link Request}, which is then put onto the server's {@link ArrayBlockingQueue}. */ - private void send(RequestType command, int subTaskIndex) { + private void send(RequestType command, int subtaskIndex) { log.debug("Connected to subtask server, sending request"); - Request request = new Request(command, subTaskIndex, this); + Request request = new Request(command, subtaskIndex, this); SubtaskServer.submitRequest(request); } @@ -124,4 +133,8 @@ private Response receive() { return null; } } + + long tryAgainWaitTimeMillis() { + return TRY_AGAIN_WAIT_TIME_MILLIS; + } } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskDirectoryIterator.java b/src/main/java/gov/nasa/ziggy/module/SubtaskDirectoryIterator.java index 2551e58..e1a0294 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskDirectoryIterator.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskDirectoryIterator.java @@ -24,7 +24,7 @@ */ public class SubtaskDirectoryIterator implements Iterator { private static final Logger log = LoggerFactory.getLogger(SubtaskDirectoryIterator.class); - private static final Pattern SUB_TASK_PATTERN = Pattern.compile("st-([0-9]+)"); + private static final Pattern SUBTASK_PATTERN = Pattern.compile("st-([0-9]+)"); private final Iterator dirIterator; private final LinkedList directoryList; @@ -33,8 +33,8 @@ public class SubtaskDirectoryIterator implements Iterator public SubtaskDirectoryIterator(File taskDirectory) { directoryList = new LinkedList<>(); buildDirectoryList(taskDirectory); - log.debug("Number of subtask directories detected in task directory " - + taskDirectory.toString() + ": " + directoryList.size()); + log.debug("Detected {} subtask directories in task directory {}", taskDirectory.toString(), + directoryList.size()); dirIterator = directoryList.iterator(); } @@ -45,7 +45,7 @@ private void buildDirectoryList(File dir) { File groupDir = file.getParentFile(); File subtaskDir = file; directoryList.add(new GroupSubtaskDirectory(groupDir, subtaskDir)); - log.debug("Adding: " + file); + log.debug("Adding {}", file); } } @@ -78,7 +78,7 @@ private List listSubtaskDirsNumericallyOrdered(File dir) { } private int subtaskNumber(String name) { - Matcher matcher = SUB_TASK_PATTERN.matcher(name); + Matcher matcher = SUBTASK_PATTERN.matcher(name); int number = -1; if (matcher.matches()) { number = Integer.parseInt(matcher.group(1)); @@ -90,7 +90,7 @@ public int getCurrentIndex() { return currentIndex; } - public int numSubTasks() { + public int numSubtasks() { return directoryList.size(); } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java b/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java index 7796ca0..10eb227 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java @@ -43,7 +43,7 @@ /** * This class encapsulates the setup and execution algorithm code for a single subtask. The code is * assumed to be runnable from the shell using the binary name for invocation. It also invokes the - * populateSubTaskInputs() method of the module's {@link PipelineInputs} subclass prior to running + * populateSubtaskInputs() method of the module's {@link PipelineInputs} subclass prior to running * the algorithm, and invokes the populateTaskResults() method of the module's * {@link PipelineOutputs} subclass subsequent to running the algorithm. *

@@ -71,7 +71,7 @@ public class SubtaskExecutor { private CommandLine commandLine; private Map environment = new HashMap<>(); - private OperatingSystemType osType = OperatingSystemType.getInstance(); + private OperatingSystemType osType = OperatingSystemType.newInstance(); private String libPath; private String binPath; @@ -117,11 +117,11 @@ private void initialize() { String hostname = HostNameUtils.shortHostName(); - log.info("osType = " + osType.toString()); - log.info("hostname = " + hostname); - log.info("binaryDir = " + binaryDir); - log.info("binaryName = " + binaryName); - log.info("libPath = " + libPath); + log.info("osType = {}", osType.toString()); + log.info("hostname = {}", hostname); + log.info("binaryDir = {}", binaryDir); + log.info("binaryName = {}", binaryName); + log.info("libPath = {}", libPath); // Construct the environment environment.put(MCR_CACHE_ROOT_ENV_VAR_NAME, @@ -153,7 +153,7 @@ private void initialize() { */ private File binaryDir(String binPathString, String binaryName) { File binFile = binaryDirInternal(binPathString, binaryName, null); - if (binFile == null && OperatingSystemType.getInstance() == OperatingSystemType.MAC_OS_X) { + if (binFile == null && OperatingSystemType.newInstance() == OperatingSystemType.MAC_OS_X) { binFile = binaryDirInternal(binPathString, binaryName, new String[] { binaryName + ".app", "Contents", "MacOS" }); } @@ -162,7 +162,7 @@ private File binaryDir(String binPathString, String binaryName) { private File binaryDirInternal(String binPathString, String binaryName, String[] pathSuffix) { - log.info("Searching for binary " + binaryName + " in path " + binPathString); + log.info("Searching for binary {} in path {}", binaryName, binPathString); File binFile = null; String[] binPaths = binPathString.split(File.pathSeparator); for (String binPath : binPaths) { @@ -238,7 +238,7 @@ private void populateEnvironmentFromPropertiesFile() { } sb.setLength(sb.length() - 2); sb.append("]"); - log.info("Execution environment: " + sb.toString()); + log.info("Execution environment is {}", sb.toString()); } /** @@ -265,16 +265,16 @@ public int execAlgorithm() { retCode = execAlgorithmInternal(); if (retCode != 0) { - log.warn("Marking subtask as failed because retCode = " + retCode); + log.warn("Marking subtask as failed (retCode={})", retCode); markSubtaskFailed(workingDir); } if (errorFile.exists()) { - log.warn("Marking subtask as failed because an error file exists"); + log.warn("Marking subtask as failed (error file exists)"); markSubtaskFailed(workingDir); } } catch (Exception e) { - log.warn("Marking subtask as failed because a Java-side exception occurred", e); + log.warn("Marking subtask as failed (Java-side exception occurred)", e); markSubtaskFailed(workingDir); } return retCode; @@ -292,7 +292,7 @@ public int execAlgorithmInternal() { } AlgorithmStateFiles stateFile = new AlgorithmStateFiles(workingDir); - stateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.PROCESSING); + stateFile.updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); boolean inputsProcessingSucceeded = false; boolean algorithmProcessingSucceeded = false; @@ -320,7 +320,7 @@ public int execAlgorithmInternal() { File errorFile = ModuleInterfaceUtils.errorFile(workingDir, binaryName); if (retCode == 0 && !errorFile.exists()) { - stateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + stateFile.updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); } else { /* * Don't handle an error in processing at this point in execution. Instead, allow the @@ -328,15 +328,15 @@ public int execAlgorithmInternal() { * level, after some error-management tasks have been completed. */ - stateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); + stateFile.updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); if (retCode != 0) { if (!inputsProcessingSucceeded) { - log.error("failed to generate sub-task inputs, retCode = " + retCode); + log.error("Failed to generate subtask inputs (retCode={})", retCode); } else if (algorithmProcessingSucceeded) { - log.error("failed to generate task results, retCode = " + retCode); + log.error("Failed to generate task results (retCode={})", retCode); } } else { - log.info("Algorithm process completed, retCode=" + retCode); + log.info("Algorithm process completed (retCode={})", retCode); } } @@ -349,7 +349,7 @@ public int execAlgorithmInternal() { */ public int execSimple(List commandLineArgs) { int retCode = runCommandline(commandLineArgs, binaryName); - log.info("execSimple: retCode = " + retCode); + log.info("retCode={}", retCode); return retCode; } @@ -406,7 +406,7 @@ int runInputsOutputsCommand(Class inputsOutputsClass) { throw new UncheckedIOException("Unable to get process environment ", e); } - log.info("Executing command: " + commandLine.toString()); + log.info("Executing command {}", commandLine.toString()); return externalProcess.execute(); } @@ -423,19 +423,19 @@ int runCommandline(List commandline, String logPrefix) { ZiggyFileUtils.ZIGGY_CHARSET)) { File binary = new File(binaryDir.getPath(), binaryName); if ((!binary.exists() || !binary.isFile()) - && OperatingSystemType.getInstance() == OperatingSystemType.MAC_OS_X) { + && OperatingSystemType.newInstance() == OperatingSystemType.MAC_OS_X) { binary = new File(binaryDir.getPath(), binaryName + ".app/Contents/MacOS/" + binaryName); } - log.info("executing " + binary); + log.info("binary={}", binary); commandLine = new CommandLine(binary.getCanonicalPath()); for (String element : commandline) { commandLine.addArgument(element); } - log.info("CommandLine: " + commandLine); + log.info("commandLine={}", commandLine); Map env = EnvironmentUtils.getProcEnvironment(); @@ -480,7 +480,7 @@ int runCommandline(List commandline, String logPrefix) { externalProcess.timeout(timeoutSecs * 1000); externalProcess.setCommandLine(commandLine); - log.info("env = " + env); + log.info("env={}", env); retCode = externalProcess.execute(); } finally { IntervalMetric.stop("pipeline.module.externalProcess." + binaryName + ".execTime", @@ -502,9 +502,9 @@ private Map mergeWithEnvironment(Map additionalE } private static void markSubtaskFailed(File workingDir) { - AlgorithmStateFiles subTaskState = new AlgorithmStateFiles(workingDir); - if (subTaskState.currentSubtaskState() != AlgorithmStateFiles.SubtaskState.FAILED) { - subTaskState.updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); + AlgorithmStateFiles subtaskState = new AlgorithmStateFiles(workingDir); + if (subtaskState.currentAlgorithmState() != AlgorithmStateFiles.AlgorithmState.FAILED) { + subtaskState.updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); } } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java b/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java index 6f213a4..a5e557d 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java @@ -5,7 +5,7 @@ import java.io.UncheckedIOException; import java.nio.file.Paths; import java.util.Objects; -import java.util.concurrent.Semaphore; +import java.util.concurrent.CountDownLatch; import org.apache.commons.exec.DefaultExecutor; import org.apache.commons.lang3.StringUtils; @@ -16,7 +16,6 @@ import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; import gov.nasa.ziggy.module.io.AlgorithmErrorReturn; import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; -import gov.nasa.ziggy.module.remote.TimestampFile; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.io.LockManager; @@ -44,18 +43,18 @@ public class SubtaskMaster implements Runnable { int threadNumber = -1; private final String node; - private final Semaphore complete; + private final CountDownLatch countdownLatch; private final String binaryName; private final String taskDir; private final int timeoutSecs; private final String jobId; private final String jobName; - public SubtaskMaster(int threadNumber, String node, Semaphore complete, String binaryName, - String taskDir, int timeoutSecs) { + public SubtaskMaster(int threadNumber, String node, CountDownLatch countdownLatch, + String binaryName, String taskDir, int timeoutSecs) { this.threadNumber = threadNumber; this.node = node; - this.complete = complete; + this.countdownLatch = countdownLatch; this.binaryName = binaryName; this.taskDir = taskDir; this.timeoutSecs = timeoutSecs; @@ -64,8 +63,7 @@ public SubtaskMaster(int threadNumber, String node, Semaphore complete, String b if (!StringUtils.isBlank(fullJobId)) { jobId = fullJobId.split("\\.")[0]; jobName = System.getenv("PBS_JOBNAME"); - log.info( - "job ID: " + jobId + ", job name: " + jobName + ", thread number: " + threadNumber); + log.info("jobId={}, jobName={}, threadNumber={}", jobId, jobName, threadNumber); } else { jobId = "none"; jobName = "none"; @@ -77,12 +75,12 @@ public SubtaskMaster(int threadNumber, String node, Semaphore complete, String b public void run() { try { processSubtasks(); - log.info("Node: " + node + "[" + threadNumber - + "]: No more subtasks to process, thread exiting"); + log.info("Node: {}[{}]: No more subtasks to process, thread exiting", node, + threadNumber); } catch (Exception e) { log.error("Exception thrown in SubtaskMaster", e); } finally { - complete.release(); + countdownLatch.countDown(); } } @@ -103,7 +101,7 @@ private void processSubtasks() { response = subtaskClient.nextSubtask(); if (response == null) { - log.error("Null response from SubtaskClient, exiting."); + log.error("Null response from SubtaskClient, exiting"); break; } if (response.status.equals(ResponseType.NO_MORE)) { @@ -111,13 +109,13 @@ private void processSubtasks() { break; } if (!response.successful()) { - log.error("Unsuccessful response from SubtaskClient, exiting."); - log.error("Response content: {}", response.toString()); + log.error("Unsuccessful response from SubtaskClient, exiting"); + log.error("Response is {}", response.toString()); break; } subtaskIndex = response.subtaskIndex; - log.debug(threadNumber + ": Processing sub-task: " + subtaskIndex); + log.debug("threadNumber={}, subtaskIndex={}", threadNumber, subtaskIndex); File subtaskDir = SubtaskUtils.subtaskDirectory(Paths.get(taskDir), subtaskIndex) .toFile(); @@ -129,8 +127,8 @@ private void processSubtasks() { SubtaskUtils.putLogStreamIdentifier(subtaskDir); if (!checkSubtaskState(subtaskDir)) { executeSubtask(subtaskDir, threadNumber, subtaskIndex); - subtaskClient.reportSubtaskComplete(subtaskIndex); } + subtaskClient.reportSubtaskComplete(subtaskIndex); } else { subtaskClient.reportSubtaskLocked(subtaskIndex); } @@ -141,6 +139,10 @@ private void processSubtasks() { // here to prevent same. The higher-level monitoring will manage any // cases in which a subtask's processing failed. logException(subtaskIndex, e); + + // Also, tell the server and allocator not to bother trying again with + // this subtask. + subtaskClient.reportSubtaskComplete(subtaskIndex); } finally { SubtaskUtils.putLogStreamIdentifier((String) null); if (lockFileObtained) { @@ -171,31 +173,30 @@ private void processSubtasks() { private boolean checkSubtaskState(File subtaskDir) { AlgorithmStateFiles previousAlgorithmState = algorithmStateFiles(subtaskDir); - if (!previousAlgorithmState.subtaskStateExists()) { + if (!previousAlgorithmState.stateExists()) { // no previous run exists - log.info("No previous algorithm state file found in " + subtaskDir.getName() - + ", executing this subtask"); + log.info("No previous algorithm state file found in {}, executing this subtask", + subtaskDir.getName()); return false; } if (previousAlgorithmState.isComplete()) { - log.info("subtask algorithm state = COMPLETE, skipping subtask" + subtaskDir.getName()); + log.info("Subtask algorithm state COMPLETE, skipping subtask{}", subtaskDir.getName()); return true; } if (previousAlgorithmState.isFailed()) { - log.info(".FAILED state detected in directory " + subtaskDir.getName()); + log.info(".FAILED state detected in directory {}", subtaskDir.getName()); return true; } if (previousAlgorithmState.isProcessing()) { - log.info(".PROCESSING state detected in directory " + subtaskDir.getName()); + log.info(".PROCESSING state detected in directory {}", subtaskDir.getName()); return true; } - log.info( - "Unexpected subtask algorithm state = " + previousAlgorithmState.currentSubtaskState() - + ", restarting subtask " + subtaskDir.getName()); + log.info("Unexpected subtask algorithm state {}, restarting subtask {}", + previousAlgorithmState.currentAlgorithmState(), subtaskDir.getName()); return false; } @@ -212,18 +213,18 @@ private void executeSubtask(File subtaskDir, int threadNumber, int subtaskIndex) + ".node." + node; try { new File(subtaskDir, jobInfoFileName).createNewFile(); - TimestampFile.create(subtaskDir, TimestampFile.Event.SUB_TASK_START); + TimestampFile.create(subtaskDir, TimestampFile.Event.SUBTASK_START); SubtaskExecutor subtaskExecutor = subtaskExecutor(subtaskIndex); - log.info("START subtask: " + subtaskIndex + " on " + node + "[" + threadNumber + "]"); + log.info("START subtask {} on {}[{}]", subtaskIndex, node, threadNumber); retCode = subtaskExecutor.execAlgorithm(); - log.info("FINISH subtask " + subtaskIndex + " on " + node + ", rc: " + retCode); + log.info("FINISH subtask {} on {} (retCode={})", subtaskIndex, node, retCode); } catch (IOException e) { throw new UncheckedIOException( "Unable to create file " + new File(subtaskDir, jobInfoFileName).toString(), e); } finally { - TimestampFile.create(subtaskDir, TimestampFile.Event.SUB_TASK_FINISH); + TimestampFile.create(subtaskDir, TimestampFile.Event.SUBTASK_FINISH); } if (retCode != 0) { @@ -269,7 +270,7 @@ AlgorithmStateFiles algorithmStateFiles(File subtaskDir) { * support testing. */ void logException(int subtaskIndex, Exception e) { - log.error("Error occurred during processing of subtask " + subtaskIndex, e); + log.error("Error occurred during processing of subtask {}", subtaskIndex, e); } /** Writes the algorithm stack trace, if any, to the algorithm log. */ diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java b/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java index ace6197..5c3b7e8 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java @@ -10,7 +10,7 @@ import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** - * Serves sub-tasks to clients using {@link SubtaskAllocator}. Clients should use + * Serves subtasks to clients using {@link SubtaskAllocator}. Clients should use * {@link SubtaskClient} to communicate with an instance of this class. * * @author Todd Klaus @@ -135,9 +135,9 @@ public Response(ResponseType status) { this.status = status; } - public Response(ResponseType status, int subTaskIndex) { + public Response(ResponseType status, int subtaskIndex) { this.status = status; - subtaskIndex = subTaskIndex; + this.subtaskIndex = subtaskIndex; } public boolean successful() { @@ -149,7 +149,7 @@ public String toString() { StringBuilder sb = new StringBuilder(); sb.append("Response [status="); sb.append(status); - sb.append(", subTaskIndex="); + sb.append(", subtaskIndex="); sb.append(subtaskIndex); sb.append("]"); @@ -176,7 +176,7 @@ public void run() { // Retrieve the next request, or block until one is provided. Request request = requestQueue.take(); - log.debug("listen[server,before]: request: " + request); + log.debug("listen[server,before] request={}", request); Response response = null; @@ -185,7 +185,7 @@ public void run() { if (type == RequestType.GET_NEXT) { SubtaskAllocation nextSubtask = subtaskAllocator().nextSubtask(); - log.debug("Allocated: " + nextSubtask); + log.debug("Allocated {}", nextSubtask); ResponseType status = nextSubtask.getStatus(); int subtaskIndex = nextSubtask.getSubtaskIndex(); @@ -201,10 +201,10 @@ public void run() { subtaskAllocator().markSubtaskLocked(request.subtaskIndex); response = new Response(ResponseType.OK); } else { - log.error("Unknown command: " + type); + log.error("Unknown command {}", type); } - log.debug("listen[server,after], response: " + response); + log.debug("listen[server,after] response={}", response); // Send the response back to the client. request.client.submitResponse(response); diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java b/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java index 41985f1..0aaf4ef 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java @@ -68,6 +68,8 @@ public static void putLogStreamIdentifier(String logStreamIdentifier) { } public static void clearStaleAlgorithmStates(File taskDir) { + log.info("Removing stale PROCESSING state from task directory"); + new AlgorithmStateFiles(taskDir).clearStaleState(); log.info("Finding and clearing stale PROCESSING or FAILED subtask states"); SubtaskDirectoryIterator it = new SubtaskDirectoryIterator(taskDir); while (it.hasNext()) { diff --git a/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java b/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java index 95d20d4..260e980 100644 --- a/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java +++ b/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java @@ -24,6 +24,7 @@ public class TaskDirectoryManager { private final Path taskDataDir; private final PipelineTask pipelineTask; + private Path taskDir; public TaskDirectoryManager(PipelineTask pipelineTask) { taskDataDir = DirectoryProperties.taskDataDir(); @@ -31,23 +32,21 @@ public TaskDirectoryManager(PipelineTask pipelineTask) { } public Path taskDir() { - return taskDir(pipelineTask.taskBaseName()); - } - - private Path taskDir(String taskBaseName) { - return taskDataDir.resolve(taskBaseName); + if (taskDir == null) { + taskDir = taskDataDir.resolve(pipelineTask.taskBaseName()); + } + return taskDir; } @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public synchronized Path allocateTaskDir(boolean cleanExisting) { if (Files.isDirectory(taskDir()) && cleanExisting) { - log.info( - "Working directory for name=" + pipelineTask.getId() + " already exists, deleting"); + log.info("Working directory for task {} already exists, deleting", pipelineTask); ZiggyFileUtils.deleteDirectoryTree(taskDir()); } - log.info("Creating task working dir: " + taskDir().toString()); + log.info("Creating task working dir {}", taskDir().toString()); try { Files.createDirectories(taskDir()); } catch (IOException e) { diff --git a/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java b/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java index c39d9e2..dee2260 100644 --- a/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java +++ b/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java @@ -3,58 +3,151 @@ import java.io.File; import java.nio.file.Path; import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.module.AlgorithmStateFiles.SubtaskState; +import gov.nasa.ziggy.module.AlgorithmStateFiles.AlgorithmState; import gov.nasa.ziggy.module.AlgorithmStateFiles.SubtaskStateCounts; -import gov.nasa.ziggy.module.StateFile.State; -import gov.nasa.ziggy.util.io.LockManager; +import gov.nasa.ziggy.module.TimestampFile.Event; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.services.messages.AllJobsFinishedMessage; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; +import gov.nasa.ziggy.services.messages.TaskProcessingCompleteMessage; +import gov.nasa.ziggy.services.messages.WorkerStatusMessage; +import gov.nasa.ziggy.services.messaging.ZiggyMessenger; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.util.ZiggyUtils; /** * Provides tools to manage a task's state file. *

* The main function of the class is to, upon request, walk through the subtask directories for a - * given task and count the number of failed and completed subtasks. This information is used in two - * ways: - *

    - *
  1. It allows the compute nodes to determine whether all subtasks are through with processing, - * which is important to execution decisions made by {@link ComputeNodeMaster}. - *
  2. It allows the state file's subtask counts to be updated, and if necessary it allows the state - * to be updated to reflect that errors have occurred. - *
- * In addition, once the {@link ComputeNodeMaster} has completed its post-processing, the class - * allows the state file to be marked as complete. + * given task and count the number of failed and completed subtasks. This allows the subtask counts + * in the database to be updated. + *

+ * The TaskMonitor also detects whether all subtasks for a task are either completed or failed, and + * notifies the {@link AlgorithmMonitor} of this fact via an instance of + * {@link TaskProcessingCompleteMessage}. Conversely, the TaskMonitor also responds to + * {@link AllJobsFinishedMessage} instances sent from the {@link AlgorithmMonitor} by shutting down + * the monitoring for the given task and performing a final count of subtask states. The + * {@link AllJobsFinishedMessage} means that all remote jobs for the task have finished or been + * deleted, ergo no further processing will occur regardless of how many subtasks have not yet been + * processed. * * @author PT + * @author Bill Wohler */ -public class TaskMonitor { +public class TaskMonitor implements Runnable { private static final Logger log = LoggerFactory.getLogger(TaskMonitor.class); - private final StateFile stateFile; + private static final long PROCESSING_MESSAGE_MAX_WAIT_MILLIS = 5000L; + private static final long FILE_SYSTEM_LAG_DELAY_MILLIS = 5000L; + private static final long FILE_SYSTEM_CHECK_INTERVAL_MILLIS = 100L; + private static final int FILE_SYSTEM_CHECKS_COUNT = (int) (FILE_SYSTEM_LAG_DELAY_MILLIS + / FILE_SYSTEM_CHECK_INTERVAL_MILLIS); + private final File taskDir; - private final File lockFile; private final List subtaskDirectories; + private final ScheduledThreadPoolExecutor monitoringThread = new ScheduledThreadPoolExecutor(1); + final long pollIntervalMilliseconds; + final AlgorithmStateFiles taskAlgorithmStateFile; + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private final PipelineTask pipelineTask; + private int totalSubtasks; + private boolean monitoringEnabled = true; + private boolean finishFileDetected; - public TaskMonitor(StateFile stateFile, File taskDir) { + public TaskMonitor(PipelineTask pipelineTask, File taskDir, long pollIntervalMilliseconds) { subtaskDirectories = SubtaskUtils.subtaskDirectories(taskDir.toPath()); - this.stateFile = stateFile; this.taskDir = taskDir; - lockFile = new File(taskDir, StateFile.LOCK_FILE_NAME); + this.pollIntervalMilliseconds = pollIntervalMilliseconds; + taskAlgorithmStateFile = new AlgorithmStateFiles(taskDir); + this.pipelineTask = pipelineTask; + } + + public void startMonitoring() { + if (pollIntervalMilliseconds > 0) { + monitoringThread.scheduleWithFixedDelay(this, 0, pollIntervalMilliseconds, + TimeUnit.MILLISECONDS); + ZiggyShutdownHook.addShutdownHook(() -> { + monitoringThread.shutdownNow(); + }); + } + + // If the worker for this task has sent a final message, perform a final + // update and shut down the monitoring thread. + ZiggyMessenger.subscribe(WorkerStatusMessage.class, message -> { + handleWorkerStatusMessage(message); + }); + + // If all remote jobs for this task have exited, perform a final update and + // shut down the monitoring thread. + ZiggyMessenger.subscribe(AllJobsFinishedMessage.class, message -> { + handleAllJobsFinishedMessage(message); + }); + + // If the task has been halted, perform a final update and shut down the + // monitoring thread. + ZiggyMessenger.subscribe(HaltTasksRequest.class, message -> { + handleHaltTasksRequest(message); + }); + } + + @Override + public void run() { + update(); + } + + void handleWorkerStatusMessage(WorkerStatusMessage message) { + + // We only care about the worker message if we're doing local processing; otherwise, the + // worker's exit is irrelevant because processing has already been handed over to PBS. + if (message.isLastMessageFromWorker() && message.getPipelineTask().equals(pipelineTask) + && pipelineTaskDataOperations.algorithmType(pipelineTask) == AlgorithmType.LOCAL) { + update(true); + } + } + + void handleAllJobsFinishedMessage(AllJobsFinishedMessage message) { + if (!pipelineTask.equals(message.getPipelineTask())) { + return; + } + update(true); + } + + void handleHaltTasksRequest(HaltTasksRequest request) { + if (request.getPipelineTasks().contains(pipelineTask)) { + update(true); + } } private SubtaskStateCounts countSubtaskStates() { + + // If this is the first time we're counting states, make sure that the total subtask + // count is set correctly. + if (totalSubtasks == 0) { + totalSubtasks = pipelineTaskDataOperations.subtaskCounts(pipelineTask) + .getTotalSubtaskCount(); + } + SubtaskStateCounts stateCounts = new SubtaskStateCounts(); if (subtaskDirectories.isEmpty()) { - log.warn("No subtask directories found in: " + taskDir); + log.warn("No subtask directories found in {}", taskDir); } for (Path subtaskDir : subtaskDirectories) { AlgorithmStateFiles currentSubtaskStateFile = new AlgorithmStateFiles( subtaskDir.toFile()); - SubtaskState currentSubtaskState = currentSubtaskStateFile.currentSubtaskState(); + AlgorithmState currentSubtaskState = currentSubtaskStateFile.currentAlgorithmState(); if (currentSubtaskState == null) { // no algorithm state file exists yet @@ -66,6 +159,10 @@ private SubtaskStateCounts countSubtaskStates() { return stateCounts; } + public boolean allSubtasksProcessed() { + return allSubtasksProcessed(countSubtaskStates()); + } + /** * Determines whether all subtasks have been processed: specifically, this means that all the * subtasks are in either the completed or failed states, and none are currently processing or @@ -73,82 +170,142 @@ private SubtaskStateCounts countSubtaskStates() { * * @return true if all subtasks have been processed. */ - public boolean allSubtasksProcessed() { - SubtaskStateCounts stateCounts = countSubtaskStates(); - return stateCounts.getCompletedSubtasks() + stateCounts.getFailedSubtasks() == stateFile - .getNumTotal(); + public boolean allSubtasksProcessed(SubtaskStateCounts stateCounts) { + return stateCounts.getCompletedSubtasks() + + stateCounts.getFailedSubtasks() == totalSubtasks; } /** - * Makes a single pass through all of the subtask directories and updates the {@link StateFile} - * based on the {@link AlgorithmStateFiles}s. This method does not update the status to COMPLETE - * when all subtasks are done to allow the caller to do any post processing before the state - * file is updated. The state file should be marked COMPLETE with the markStateFileDone() - * method. + * Makes a single pass through all of the subtask directories and updates the database based on + * the {@link AlgorithmStateFiles} instances. If the task has started processing, the update + * will detect the .PROCESSING file in the task directory and set the task step to EXECUTING. */ - public void updateState() { - try { - LockManager.getWriteLockOrBlock(lockFile); - StateFile diskStateFile = stateFile.newStateFileFromDiskFile(true); - - if (subtaskDirectories.isEmpty()) { - log.warn("No subtask dirs found in: " + taskDir); - } - - SubtaskStateCounts stateCounts = countSubtaskStates(); - stateFile.setNumComplete(stateCounts.getCompletedSubtasks()); - stateFile.setNumFailed(stateCounts.getFailedSubtasks()); - stateFile.setState(diskStateFile.getState()); - // If for some reason this state hasn't been upgraded to PROCESSING, - // do that now. - stateFile - .setState(diskStateFile.isStarted() ? diskStateFile.getState() : State.PROCESSING); - updateStateFile(diskStateFile); - } finally { - LockManager.releaseWriteLock(lockFile); - } + public void update() { + update(false); } /** - * Move the {@link StateFile} into the completed state if all subtasks are complete, or into the - * failed state if some subtasks failed or were never processed. + * Performs a task status update. If argument finalUpdate is true, the {@link TaskMonitor} + * performs an orderly shutdown of monitoring; this occurs in response to messages received by + * the task monitor, as all such messages signal to the monitor that processing has ended. The + * method is synchronized in order to prevent the monitoring loop and the message-induced update + * from interfering with one another. */ - public void markStateFileDone() { + private synchronized void update(boolean finalUpdate) { - try { - LockManager.getWriteLockOrBlock(lockFile); + // If we're no longer monitoring, it means we don't need this update. + if (!monitoringEnabled) { + return; + } - StateFile previousStateFile = stateFile.newStateFileFromDiskFile(); + if (subtaskDirectories.isEmpty()) { + log.warn("No subtask dirs found in {}", taskDir); + } - if (stateFile.getNumComplete() + stateFile.getNumFailed() == stateFile.getNumTotal()) { - log.info("All subtasks complete or errored, marking state file COMPLETE"); - } else { - // If there is a shortfall, consider the missing sub-tasks failed - int missing = stateFile.getNumTotal() - - (stateFile.getNumComplete() + stateFile.getNumFailed()); + SubtaskStateCounts stateCounts = countSubtaskStates(); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, -1, + stateCounts.getCompletedSubtasks(), stateCounts.getFailedSubtasks()); - log.info("Missing subtasks, forcing state to FAILED, missing=" + missing); - stateFile.setNumFailed(stateFile.getNumFailed() + missing); - } - stateFile.setState(StateFile.State.COMPLETE); + if (taskAlgorithmStateFile.isProcessing() + && pipelineTaskDataOperations.processingStep(pipelineTask).isPreExecutionStep()) { + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.EXECUTING); + } + + boolean allSubtasksProcessed = allSubtasksProcessed(stateCounts); + + // If this was a run-of-the-mill update, we're done. + if (!allSubtasksProcessed && !finalUpdate) { + return; + } - updateStateFile(previousStateFile); - } finally { - LockManager.releaseWriteLock(lockFile); + if (finalUpdate) { + log.debug("Final update for task {}", pipelineTask.getId()); } + if (allSubtasksProcessed) { + log.debug("All subtasks processed for task {}", pipelineTask.getId()); + } + + // If we got this far, then all subsequent calls to this method should return + // without taking any action. + monitoringEnabled = false; + checkForFinishFile(); + publishTaskProcessingCompleteMessage(allSubtasksProcessed ? new CountDownLatch(1) : null); + + // It is now safe to shut down the monitoring loop. + shutdown(); } - private void updateStateFile(StateFile previousStateFile) { - if (!previousStateFile.equals(stateFile)) { - log.info("Updating state: " + previousStateFile + " -> " + stateFile); + /** + * Checks for the FINISH timestamp file in the task directory. This is created by the + * {@link ComputeNodeMaster} when it exits and is needed by + * {@link ExternalProcessPipelineModule} when it persists results. Because the subtask + * .COMPLETED files can appear before the compute node FINISH file, and because there are file + * system lags on network file systems, we need to do a repetitive check for the file rather + * than just a one-and-done. + */ + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + private void checkForFinishFile() { + try { + ZiggyUtils.tryPatiently("Wait for FINISH file", fileSystemChecksCount(), + fileSystemCheckIntervalMillis(), () -> { + if (!TimestampFile.exists(taskDir, Event.FINISH)) { + throw new Exception(); + } + finishFileDetected = true; + return null; + }); + } catch (PipelineException e) { + log.error("FINISH file never created in task directory {}", taskDir.toString()); + finishFileDetected = false; + } + } - if (!StateFile.updateStateFile(previousStateFile, stateFile)) { - log.error("Failed to update state file: " + previousStateFile); + public void shutdown() { + monitoringThread.shutdown(); + } + + void publishTaskProcessingCompleteMessage(CountDownLatch processingCompleteMessageLatch) { + ZiggyMessenger.publish(new TaskProcessingCompleteMessage(pipelineTask), false, + processingCompleteMessageLatch); + if (processingCompleteMessageLatch != null) { + try { + processingCompleteMessageLatch.await(PROCESSING_MESSAGE_MAX_WAIT_MILLIS, + TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); } } } - public StateFile getStateFile() { - return stateFile; + List getSubtaskDirectories() { + return subtaskDirectories; + } + + Path getTaskDir() { + return taskDir.toPath(); + } + + AlgorithmStateFiles getTaskAlgorithmStateFile() { + return taskAlgorithmStateFile; + } + + long fileSystemCheckIntervalMillis() { + return FILE_SYSTEM_CHECK_INTERVAL_MILLIS; + } + + int fileSystemChecksCount() { + return FILE_SYSTEM_CHECKS_COUNT; + } + + boolean isFinishFileDetected() { + return finishFileDetected; + } + + void resetFinishFileDetection() { + finishFileDetected = false; + } + + void resetMonitoringEnabled() { + monitoringEnabled = true; } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/TimestampFile.java b/src/main/java/gov/nasa/ziggy/module/TimestampFile.java similarity index 65% rename from src/main/java/gov/nasa/ziggy/module/remote/TimestampFile.java rename to src/main/java/gov/nasa/ziggy/module/TimestampFile.java index ebab2aa..0ccd098 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/TimestampFile.java +++ b/src/main/java/gov/nasa/ziggy/module/TimestampFile.java @@ -1,15 +1,18 @@ -package gov.nasa.ziggy.module.remote; +package gov.nasa.ziggy.module; import java.io.File; import java.io.FileFilter; import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Set; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.ZiggyFileUtils; /** * @author Todd Klaus @@ -18,7 +21,7 @@ public abstract class TimestampFile { private static final Logger log = LoggerFactory.getLogger(TimestampFile.class); public enum Event { - ARRIVE_PFE, QUEUED_PBS, PBS_JOB_START, PBS_JOB_FINISH, SUB_TASK_START, SUB_TASK_FINISH + ARRIVE_COMPUTE_NODES, QUEUED, START, FINISH, SUBTASK_START, SUBTASK_FINISH } @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) @@ -41,20 +44,32 @@ public static boolean create(File directory, Event name, long timestamp) { } } - public static boolean delete(File directory, Event name) { - // delete any existing files with this prefix - String prefix = name.toString(); + public static boolean exists(File directory, Event name) { + return !find(directory, name).isEmpty(); + } + + public static Set find(File directory, Event name) { + return ZiggyFileUtils.listFiles(directory.toPath(), pattern(name)); + } - File[] files = directory.listFiles(); - for (File file : files) { - if (file.getName().startsWith(prefix)) { - boolean deleted = file.delete(); + private static String pattern(Event name) { + return name.toString() + "\\.[0-9]+"; + } + + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + public static boolean delete(File directory, Event name) { + Set files = find(directory, name); + for (Path file : files) { + try { + boolean deleted = Files.deleteIfExists(file); if (!deleted) { - log.warn( - String.format("failed to delete existing timestamp file, dir=%s, file=%s", - directory, file)); + log.warn("Failed to delete existing timestamp file, dir={}, file=}", directory, + file); return false; } + } catch (IOException e) { + log.error("Exception occurred when deleting {}", file.toString(), e); + return false; } } return true; @@ -64,17 +79,26 @@ public static boolean create(File directory, Event name) { return create(directory, name, System.currentTimeMillis()); } - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) public static long timestamp(File directory, final Event name) { + return timestamp(directory, name, true); + } + + @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) + public static long timestamp(File directory, final Event name, boolean errorIfMissing) { File[] files = directory .listFiles((FileFilter) f -> f.getName().startsWith(name.toString()) && f.isFile()); if (files.length == 0) { - throw new PipelineException("Found zero files that match event:" + name); + if (!errorIfMissing) { + log.warn("Unable to find {} timestamp file in directory {}", name.toString(), + directory.toString()); + return 0; + } + throw new PipelineException("Found zero files that match event: " + name); } if (files.length > 1) { - throw new PipelineException("Found more than one files that match event:" + name); + throw new PipelineException("Found more than one files that match event: " + name); } String filename = files[0].getName(); @@ -113,10 +137,9 @@ public static long elapsedTimeMillis(File directory, final Event startEvent, long finishTime = timestamp(directory, finishEvent); if (startTime == -1 || finishTime == -1) { - // at least one of the events was missing or unparsable - log.warn( - String.format("Missing or invalid timestamp files, startTime=%s, finishTime=%s", - startTime, finishTime)); + // At least one of the events was missing or unparsable. + log.warn("Missing or invalid timestamp files, startTime={}, finishTime={}", startTime, + finishTime); return 0; } return finishTime - startTime; diff --git a/src/main/java/gov/nasa/ziggy/module/WorkerMemoryManager.java b/src/main/java/gov/nasa/ziggy/module/WorkerMemoryManager.java index 4481fae..cc884f5 100644 --- a/src/main/java/gov/nasa/ziggy/module/WorkerMemoryManager.java +++ b/src/main/java/gov/nasa/ziggy/module/WorkerMemoryManager.java @@ -32,10 +32,10 @@ public class WorkerMemoryManager { private int availableMegaBytes; public WorkerMemoryManager() { - MemInfo memInfo = OperatingSystemType.getInstance().getMemInfo(); + MemInfo memInfo = OperatingSystemType.newInstance().getMemInfo(); long physicalMemoryMegaBytes = memInfo.getTotalMemoryKB() / KILO; - log.info("physicalMemoryMegaBytes: " + physicalMemoryMegaBytes); + log.info("physicalMemoryMegaBytes={}", physicalMemoryMegaBytes); availableMegaBytes = (int) physicalMemoryMegaBytes; long jvmMaxHeapMegaBytes = Runtime.getRuntime().maxMemory() / (KILO * KILO); @@ -47,8 +47,7 @@ public WorkerMemoryManager() { */ if (jvmMaxHeapMegaBytes < physicalMemoryMegaBytes / 2) { - log.info("JVM max heap size set to jvmMaxHeapMegaBytes: " + jvmMaxHeapMegaBytes); - + log.info("JVM max heap size set to {}", jvmMaxHeapMegaBytes); availableMegaBytes -= jvmMaxHeapMegaBytes; } else { /* @@ -60,8 +59,8 @@ public WorkerMemoryManager() { long jvmInUseMegaBytes = Runtime.getRuntime().totalMemory() / (KILO * KILO); if (jvmInUseMegaBytes < physicalMemoryMegaBytes / 2) { - log.info("JVM max heap size not available, using in-use bytes: jvmInUseMegaBytes: " - + jvmInUseMegaBytes); + log.info("JVM max heap size not available, using in-use heap {}", + jvmInUseMegaBytes); availableMegaBytes -= jvmInUseMegaBytes; } else { log.info("JVM heap size not available, not accounted for in pool"); @@ -80,11 +79,10 @@ public WorkerMemoryManager(int availableMegaBytes) { private void initSemaphore() { memorySemaphore = new Semaphore(availableMegaBytes, true); - log.info("availableMegaBytes in memory manager pool: " + availableMegaBytes); + log.info("Memory manager pool has {} MB", availableMegaBytes); } /** - * @param megaBytes * @see java.util.concurrent.Semaphore#acquire(int) */ @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) @@ -96,8 +94,7 @@ public void acquireMemoryMegaBytes(int megaBytes) { try { memorySemaphore.acquire(megaBytes); - log.info( - megaBytes + " megabytes acquired, new pool size: " + availableMemoryMegaBytes()); + log.info("Acquired {} MB, new pool size is {}", megaBytes, availableMemoryMegaBytes()); } catch (InterruptedException ignored) { // If we got here, it means that a worker thread was waiting for Java heap to become // available but that thread was interrupted. It is therefore no longer waiting for @@ -106,22 +103,18 @@ public void acquireMemoryMegaBytes(int megaBytes) { } } - /** - * @param megaBytes - */ private void logAcquirePrediction(int megaBytes) { int numAvailPermits = memorySemaphore.availablePermits(); if (numAvailPermits < megaBytes) { - log.info("Requesting " + megaBytes + " megabytes from pool, but only " + numAvailPermits - + " megabytes available, " + memorySemaphore.getQueueLength() - + " threads already waiting (will probably block)..."); + log.info( + "Requesting {} MB from pool, but only {} MB available, {} threads already waiting (will probably block)", + megaBytes, numAvailPermits, memorySemaphore.getQueueLength()); } else { - log.info("Requesting " + megaBytes + " megabytes from pool (probably won't block)..."); + log.info("Requesting {} MB from pool (probably won't block)", megaBytes); } } /** - * @param megaBytes * @see java.util.concurrent.Semaphore#release(int) */ public void releaseMemoryMegaBytes(int megaBytes) { @@ -129,11 +122,11 @@ public void releaseMemoryMegaBytes(int megaBytes) { return; } - log.info("Releasing " + megaBytes + " megabytes from pool..."); + log.info("Releasing {} MB from pool", megaBytes); memorySemaphore.release(megaBytes); - log.info(megaBytes + " megabytes released, new pool size: " + availableMemoryMegaBytes()); + log.info("Released {} MB, new pool size is {} MB", megaBytes, availableMemoryMegaBytes()); } /** diff --git a/src/main/java/gov/nasa/ziggy/module/hdf5/Hdf5ModuleInterface.java b/src/main/java/gov/nasa/ziggy/module/hdf5/Hdf5ModuleInterface.java index 8750f64..88c6b87 100644 --- a/src/main/java/gov/nasa/ziggy/module/hdf5/Hdf5ModuleInterface.java +++ b/src/main/java/gov/nasa/ziggy/module/hdf5/Hdf5ModuleInterface.java @@ -97,15 +97,15 @@ static void testForUnclosedHdf5Objects(long fileId) { if (nOpen == 1) { log.info("No unclosed HDF5 objects detected"); } else { - log.warn("Number of unclosed HDF5 objects detected: " + (nOpen - 1)); - log.warn("Number of unclosed groups: " - + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_GROUP)); - log.warn("Number of unclosed datasets: " - + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_DATASET)); - log.warn("Number of unclosed datatypes: " - + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_DATATYPE)); - log.warn("Number of unclosed attributes: " - + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_ATTR)); + log.warn("Detected {} unclosed HDF5 objects", (nOpen - 1)); + log.warn(" {} unclosed groups", + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_GROUP)); + log.warn(" {} unclosed datasets", + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_DATASET)); + log.warn(" {} unclosed datatypes", + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_DATATYPE)); + log.warn(" {} unclosed attributes", + H5.H5Fget_obj_count(fileId, HDF5Constants.H5F_OBJ_ATTR)); } } diff --git a/src/main/java/gov/nasa/ziggy/module/io/AlgorithmErrorReturn.java b/src/main/java/gov/nasa/ziggy/module/io/AlgorithmErrorReturn.java index 7ab10a6..410b421 100644 --- a/src/main/java/gov/nasa/ziggy/module/io/AlgorithmErrorReturn.java +++ b/src/main/java/gov/nasa/ziggy/module/io/AlgorithmErrorReturn.java @@ -26,7 +26,7 @@ public AlgorithmErrorReturn() { } public void logStackTrace() { - log.error("Algorithm Stack Trace: msg=" + message + ", id=" + identifier); + log.error("Algorithm stack trace for msg={}, id={}", message, identifier); for (AlgorithmStack stackFrame : stack) { stackFrame.logStackTrace(); @@ -65,8 +65,7 @@ private static class AlgorithmStack implements Persistable { private int line; public void logStackTrace() { - log.error( - " Algorithm Stack Trace: file=" + file + ", name=" + name + ", line=" + line); + log.error(" Algorithm stack trace: file={}, name={}, line={}", file, name, line); } } } diff --git a/src/main/java/gov/nasa/ziggy/module/io/ModuleInterfaceUtils.java b/src/main/java/gov/nasa/ziggy/module/io/ModuleInterfaceUtils.java index 32433bc..79ffe44 100644 --- a/src/main/java/gov/nasa/ziggy/module/io/ModuleInterfaceUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/io/ModuleInterfaceUtils.java @@ -43,7 +43,7 @@ public static void writeCompanionXmlFile(Persistable inputs, String moduleName) return; } String companionXmlFile = xmlFileName(moduleName); - log.info("Writing companion xml file \"" + companionXmlFile + "\"."); + log.info("Writing companion xml file {}", companionXmlFile); StringBuilder validationErrors = new StringBuilder(); try { JAXBContext jaxbContext = JAXBContext.newInstance(inputs.getClass()); @@ -130,7 +130,7 @@ public static File errorFile(File dataDir, String filenamePrefix) { private static void deleteErrorFile(File errorFileToDelete) { boolean deleted = errorFileToDelete.delete(); if (!deleted) { - log.error("Failed to delete errorFile=" + errorFileToDelete); + log.error("Failed to delete errorFile {}", errorFileToDelete); } } @@ -143,7 +143,7 @@ public static String inputsFileName(String moduleName) { } /** - * Returns the name of the sub-task outputs file for a given module and sequence number. + * Returns the name of the subtask outputs file for a given module and sequence number. */ public static String outputsFileName(String moduleName) { return moduleName + "-outputs." + BIN_FILE_TYPE; @@ -159,14 +159,14 @@ public static Pattern outputsFileNamePattern(String moduleName) { } /** - * Returns the name of the sub-task error file for a given module and sequence number. + * Returns the name of the subtask error file for a given module and sequence number. */ public static String errorFileName(String moduleName) { return moduleName + "-error." + BIN_FILE_TYPE; } /** - * Returns the name of the sub-task XML companion file for a given module and sequence number. + * Returns the name of the subtask XML companion file for a given module and sequence number. */ public static String xmlFileName(String moduleName) { return moduleName + "-digest.xml"; diff --git a/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java b/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java index de3732c..13ef9d9 100644 --- a/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java @@ -63,7 +63,7 @@ public static String mcrPaths(String mcrRoot) { } private static OperatingSystemType osType() { - return OperatingSystemType.getInstance(); + return OperatingSystemType.newInstance(); } private static String architecture() { diff --git a/src/main/java/gov/nasa/ziggy/module/package-info.java b/src/main/java/gov/nasa/ziggy/module/package-info.java index 759ff3e..7b93af4 100644 --- a/src/main/java/gov/nasa/ziggy/module/package-info.java +++ b/src/main/java/gov/nasa/ziggy/module/package-info.java @@ -10,7 +10,7 @@ *

* The processing steps found in the {@link gov.nasa.ziggy.pipeline.definition.ProcessingStep} enum * are set with the - * {@link gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations#updateProcessingStep(long, gov.nasa.ziggy.pipeline.definition.ProcessingStep)} + * {@link gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations#updateProcessingStep(gov.nasa.ziggy.pipeline.definition.PipelineTask, gov.nasa.ziggy.pipeline.definition.ProcessingStep)} * method and implicitly with the * {@link gov.nasa.ziggy.pipeline.definition.PipelineModule#incrementProcessingStep()} method. Here * is where each of these steps occur: diff --git a/src/main/java/gov/nasa/ziggy/module/remote/PbsLogParser.java b/src/main/java/gov/nasa/ziggy/module/remote/PbsLogParser.java new file mode 100644 index 0000000..d9db08a --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/remote/PbsLogParser.java @@ -0,0 +1,112 @@ +package gov.nasa.ziggy.module.remote; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.Collection; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; + +/** + * Extracts the exit comment and exit status from one or more PBS logs. + * + * @author PT + */ +public class PbsLogParser { + + private static final Logger log = LoggerFactory.getLogger(PbsLogParser.class); + + public static final String PBS_FILE_COMMENT_PREFIX = "=>> PBS: "; + public static final String PBS_FILE_STATUS_PREFIX = "Exit Status"; + + /** + * Extracts the exit comments from a collection of PBS logs and returns in a {@link Map} with + * job ID as the map key. Jobs with no comment will have no entry in the map. + */ + public Map exitCommentByJobId( + Collection remoteJobsInformation) { + Map exitCommentByJobId = new HashMap<>(); + if (CollectionUtils.isEmpty(remoteJobsInformation)) { + return exitCommentByJobId; + } + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + String exitComment = exitComment(remoteJobInformation); + if (!StringUtils.isBlank(exitComment)) { + exitCommentByJobId.put(remoteJobInformation.getJobId(), exitComment); + } + } + return exitCommentByJobId; + } + + /** Returns the exit comment from a PBS log, or null if there is no exit comment. */ + private String exitComment(RemoteJobInformation remoteJobInformation) { + List pbsFileOutput = pbsLogFileContent(remoteJobInformation); + if (CollectionUtils.isEmpty(pbsFileOutput)) { + return null; + } + for (String pbsFileOutputLine : pbsFileOutput) { + log.debug("PBS file output line: {}", pbsFileOutputLine); + if (pbsFileOutputLine.startsWith(PBS_FILE_COMMENT_PREFIX)) { + return pbsFileOutputLine.substring(PBS_FILE_COMMENT_PREFIX.length()); + } + } + return null; + } + + /** Returns the content of a PBS log file as a {@List} of {@link String}s. */ + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + private List pbsLogFileContent(RemoteJobInformation remoteJobInformation) { + try { + return Files.readAllLines(Paths.get(remoteJobInformation.getLogFile())); + } catch (IOException e) { + // If an exception occurred, we don't want to crash the algorithm monitor, + // so return null. + return null; + } + } + + /** + * Extracts the exit status from a collection of PBS log files and returns as a {@link Map} with + * job ID as the map key. Jobs with no exit status will have no entry in the map. + */ + public Map exitStatusByJobId( + Collection remoteJobsInformation) { + Map exitStatusByJobId = new HashMap<>(); + if (CollectionUtils.isEmpty(remoteJobsInformation)) { + return exitStatusByJobId; + } + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + Integer exitStatus = exitStatus(remoteJobInformation); + if (exitStatus != null) { + exitStatusByJobId.put(remoteJobInformation.getJobId(), exitStatus); + } + } + return exitStatusByJobId; + } + + /** Returns the exit status from a PBS log file, or null if there is no exit status. */ + private Integer exitStatus(RemoteJobInformation remoteJobInformation) { + List pbsFileOutput = pbsLogFileContent(remoteJobInformation); + if (CollectionUtils.isEmpty(pbsFileOutput)) { + return null; + } + for (String pbsFileOutputLine : pbsFileOutput) { + log.debug("PBS file output line: {}", pbsFileOutputLine); + if (pbsFileOutputLine.trim().startsWith(PBS_FILE_STATUS_PREFIX)) { + int colonLocation = pbsFileOutputLine.indexOf(":"); + String returnStatusString = pbsFileOutputLine.substring(colonLocation + 1).trim(); + return Integer.parseInt(returnStatusString); + } + } + return null; + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java b/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java index 758ad47..9aac2d5 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java @@ -53,8 +53,8 @@ public void populateArchitecture(PipelineDefinitionNodeExecutionResources execut // If the architecture isn't specified, determine it via optimization. if (getArchitecture() == null) { - log.info("Selecting infrastructure for " + totalSubtasks + " subtasks using optimizer " - + executionResources.getOptimizer()); + log.info("Selecting infrastructure for {} subtasks using optimizer {}", totalSubtasks, + executionResources.getOptimizer()); selectArchitecture(executionResources, totalSubtasks, remoteCluster); } @@ -176,8 +176,9 @@ private double subtasksPerCore(PipelineDefinitionNodeExecutionResources executio if (overrideSubtasksPerCore >= subtasksPerCore) { subtasksPerCore = overrideSubtasksPerCore; } else { - log.warn("User-supplied subtasks per core " + overrideSubtasksPerCore - + " too small, using algorithmic value of " + subtasksPerCore); + log.warn( + "User-supplied subtasks per core {} too small, using algorithmic value of {}", + overrideSubtasksPerCore, subtasksPerCore); } } return subtasksPerCore; @@ -251,8 +252,8 @@ private void computeWallTimeAndQueue( RemoteQueueDescriptor descriptor = RemoteQueueDescriptor .fromQueueName(executionResources.getQueueName()); if (descriptor.equals(RemoteQueueDescriptor.UNKNOWN)) { - log.warn("Unable to determine max wall time for queue " - + executionResources.getQueueName()); + log.warn("Unable to determine max wall time for queue {}", + executionResources.getQueueName()); queueName = executionResources.getQueueName(); } else if (descriptor.equals(RemoteQueueDescriptor.RESERVED)) { queueName = executionResources.getQueueName(); diff --git a/src/main/java/gov/nasa/ziggy/module/remote/QstatMonitor.java b/src/main/java/gov/nasa/ziggy/module/remote/QstatMonitor.java deleted file mode 100644 index 5de3fb6..0000000 --- a/src/main/java/gov/nasa/ziggy/module/remote/QstatMonitor.java +++ /dev/null @@ -1,405 +0,0 @@ -package gov.nasa.ziggy.module.remote; - -import java.util.ArrayList; -import java.util.Collection; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; - -import org.apache.commons.lang3.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.module.JobMonitor; -import gov.nasa.ziggy.module.StateFile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; - -/** - * Interrogates the state of running / completed / failed jobs on a remote system via the qstat - * command line in order to identify issues with those jobs that would not otherwise be made - * apparent to pipeline operators, and also to support use of the console or picli to delete jobs - * that are queued or running. - * - * @author PT - */ -public class QstatMonitor implements JobMonitor { - - private String owner; - private String serverName; - private QueueCommandManager cmdManager; - - private static final Logger log = LoggerFactory.getLogger(QstatMonitor.class); - - // Maps from a given pipeline task name to the corresponding QstatEntry - private Map> taskNameQstatEntryMap = null; - - /** - * Constructor. Forces the QstatMonitor to get its own instance of QueueCommandManager. - */ - public QstatMonitor(String owner, String serverName) { - this(owner, serverName, null); - } - - /** - * Constructor. Uses a supplied instance of the QueueCommandManager to supply the username, - * hostname, and queue command management for this instance. - */ - public QstatMonitor(QueueCommandManager cmdManager) { - this(cmdManager.user(), cmdManager.hostname(), cmdManager); - } - - private QstatMonitor(String owner, String serverName, QueueCommandManager cmdManager) { - this.owner = owner; - this.serverName = serverName; - if (cmdManager == null) { - this.cmdManager = QueueCommandManager.newInstance(); - } else { - this.cmdManager = cmdManager; - } - taskNameQstatEntryMap = new HashMap<>(); - } - - /** - * Start monitoring the job that corresponds to a particular state file. - */ - @Override - public void addToMonitoring(StateFile stateFile) { - addToMonitoring(stateFile.taskBaseName()); - } - - @Override - public void addToMonitoring(PipelineTask pipelineTask) { - addToMonitoring(pipelineTask.taskBaseName()); - } - - private void addToMonitoring(String taskName) { - - // if the task is already in the map, we don't need to do anything else - if (!taskNameQstatEntryMap.containsKey(taskName)) { - log.info("Adding PBS job " + taskName + " to qstat monitor"); - taskNameQstatEntryMap.put(taskName, Collections.emptySet()); - } - } - - /** - * Terminate monitoring of a job that corresponds to a particular state file. - */ - @Override - public void endMonitoring(StateFile stateFile) { - log.info("Removing PBS job " + stateFile.taskBaseName() + " from qstart monitor"); - taskNameQstatEntryMap.remove(stateFile.taskBaseName()); - } - - /** - * Updates the contents of the monitor. Tasks that had not yet been assigned jobs are checked to - * see whether jobs are available. All task status values and elapsed times are updated. - */ - @Override - public void update() { - - // update the values of any jobs that already have them - updateQstatEntries(); - - // for any task that doesn't yet have a job, check for new job assignments - for (String taskName : taskNameQstatEntryMap.keySet()) { - if (taskNameQstatEntryMap.get(taskName).isEmpty()) { - Set newEntries = qstatEntriesForTask(taskName); - if (!newEntries.isEmpty()) { - log.info( - "Task " + taskName + " matched with PBS jobs " + newEntries.toString()); - } - taskNameQstatEntryMap.replace(taskName, newEntries); - } - } - } - - @Override - public Set allIncompleteJobIds(PipelineTask pipelineTask) { - return allIncompleteJobIds(pipelineTask.taskBaseName()); - } - - @Override - public Set allIncompleteJobIds(StateFile stateFile) { - return allIncompleteJobIds(stateFile.taskBaseName()); - } - - public Set allIncompleteJobIds(String taskName) { - Set qstatEntries = taskNameQstatEntryMap.get(taskName); - if (qstatEntries.isEmpty()) { - return new HashSet<>(); - } - Set ids = new TreeSet<>(); - for (QstatEntry entry : qstatEntries) { - ids.add(entry.getId()); - } - return ids; - } - - /** - * Constructs QstatEntry instances for a given job based on its name. - * - * @param taskName Name of the pipeline task. - * @return QstatEntry instance for the given job. Note that in addition to the job name, the job - * owner and the server from which it was started must match the values in the QstatMonitor. The - * return can also be null under certain circumstances, in particular the case of a job that has - * been submitted via qsub but which has not yet completed the process of queueing. + * @return - * a non-null {@link Set}. - */ - private Set qstatEntriesForTask(String taskName) { - - // get all the qstat entries, if any, for the specified task name - List qstatLines = cmdManager.getQstatInfoByTaskName(owner, taskName); - if (qstatLines.isEmpty() || qstatLines.size() == 1 && qstatLines.get(0).isBlank()) { - return Collections.emptySet(); - } - - // generate a QstatEntry for each string - List qstatEntries = qstatLines.stream() - .map(QstatEntry::new) - .collect(Collectors.toList()); - - // get the server names for all the jobs - Map jobIdToServerMap = cmdManager.serverNames( - qstatEntries.stream().map(QstatEntry::getId).sorted().collect(Collectors.toList())); - - // strip out the jobs with the wrong server names - List jobsWithCorrectServer = jobIdToServerMap.keySet() - .stream() - .filter(s -> jobIdToServerMap.get(s).equals(serverName)) - .collect(Collectors.toList()); - - // if the list is empty, return a null - if (jobsWithCorrectServer.isEmpty()) { - return Collections.emptySet(); - } - - qstatEntries = qstatEntries.stream() - .filter(s -> jobsWithCorrectServer.contains(s.getId())) - .collect(Collectors.toList()); - - // Generate a list of unique job names - List uniqueJobNames = qstatEntries.stream() - .map(QstatEntry::getName) - .distinct() - .collect(Collectors.toList()); - - Set jobIds = new HashSet<>(); - for (String name : uniqueJobNames) { - // Because of restarts, it is possible for there to be > 1 job with a given - // name, owner, and server. In this case, find the one with the max value - // of the job ID - long jobId = qstatEntries.stream() - .filter(s -> s.getName().equals(name)) - .map(QstatEntry::getId) - .max(Long::compare) - .get(); - jobIds.add(jobId); - } - - return qstatEntries.stream() - .filter(s -> jobIds.contains(s.getId())) - .collect(Collectors.toSet()); - } - - private void updateQstatEntries() { - - // construct a list of job IDs - List jobIds = new ArrayList<>(); - Set allQstatEntries = new HashSet<>(); - for (String taskName : taskNameQstatEntryMap.keySet()) { - Set qstatEntries = taskNameQstatEntryMap.get(taskName); - if (!qstatEntries.isEmpty()) { - jobIds.addAll( - qstatEntries.stream().map(QstatEntry::getId).collect(Collectors.toList())); - allQstatEntries.addAll(qstatEntries); - } - } - - if (jobIds.isEmpty()) { - return; - } - - jobIds = jobIds.stream().sorted(Long::compare).collect(Collectors.toList()); - // get the updates - Map jobNameQstatStringMap = cmdManager.getQstatInfoByJobNameMap(jobIds); - - // apply the updates - for (QstatEntry entry : allQstatEntries) { - String qstatLine = jobNameQstatStringMap.get(entry.getName()); - if (!StringUtils.isBlank(qstatLine)) { - entry.updateFromQstatString(qstatLine); - } - } - } - - /** - * Determines whether a particular task is finished based on the status of its jobs in qstat - * - * @param stateFile StateFile for the task - * @return true all jobs have status "F" indicating completion or "E" indicating exiting, false - * otherwise - */ - @Override - public boolean isFinished(StateFile stateFile) { - boolean isFinished = true; - Set entries = taskNameQstatEntryMap.get(stateFile.taskBaseName()); - if (entries.isEmpty()) { - return false; - } - for (QstatEntry entry : entries) { - String stat = entry.getStatus(); - boolean jobFinished = stat.equals("E") || stat.equals("F"); - isFinished = isFinished && jobFinished; - } - return isFinished; - } - - /** - * Retrieves the exit status for a task's jobs given its state file - * - * @param stateFile state file for the task of interest - */ - @Override - public Map exitStatus(StateFile stateFile) { - Map jobIdExitStatusMap = new HashMap<>(); - Collection jobIds = idFromStateFile(stateFile); - for (long jobId : jobIds) { - Integer exitStatus = cmdManager.exitStatus(jobId); - if (exitStatus != null) { - jobIdExitStatusMap.put(jobId, exitStatus); - } - } - return jobIdExitStatusMap; - } - - /** - * Retrieves the exit comment for a job given its state file. - * - * @param stateFile state file for the task of interest. - */ - @Override - public Map exitComment(StateFile stateFile) { - Map jobIdExitCommentMap = new HashMap<>(); - Collection jobIds = idFromStateFile(stateFile); - for (long jobId : jobIds) { - String exitComment = cmdManager.exitComment(jobId); - if (!StringUtils.isBlank(exitComment)) { - jobIdExitCommentMap.put(jobId, exitComment); - } - } - return jobIdExitCommentMap; - } - - private Set idFromStateFile(StateFile stateFile) { - String taskName = stateFile.taskBaseName(); - Set qstatEntries = taskNameQstatEntryMap.get(taskName); - return qstatEntries.stream().map(QstatEntry::getId).collect(Collectors.toSet()); - } - - // getters - @Override - public String getOwner() { - return owner; - } - - @Override - public String getServerName() { - return serverName; - } - - @Override - public QueueCommandManager getQstatCommandManager() { - return cmdManager; - } - - @Override - public Set getJobsInMonitor() { - return taskNameQstatEntryMap.keySet(); - } - - // ======================================================================================= - - /** - * Utility class that tracks useful information for a single job out of a qstat output. - * - * @author PT - */ - private static class QstatEntry { - - private final long id; - private final String name; - private String status; - - /** - * Constructs a QstatEntry instance from a line out of a qstat command. - * - * @param qstatLine Line from the output of the qstat command - */ - public QstatEntry(String qstatLine) { - - String[] qstatLineParts = qstatLine.split("\\s+"); - - // The job ID has an ID # followed by ".". - String jobId = qstatLineParts[QueueCommandManager.ID_INDEX]; - String[] jobIdParts = jobId.split("\\."); - id = Long.parseLong(jobIdParts[0]); - - // The name, owner, and status are simple strings. - status = qstatLineParts[QueueCommandManager.STATUS_INDEX]; - name = qstatLineParts[QueueCommandManager.NAME_INDEX]; - } - - /** - * Updates the status and the elapsed time from the output of a qstat command. - * - * @param qstatString Output of the qstat command. - */ - public void updateFromQstatString(String qstatString) { - String[] qstatLineParts = qstatString.split("\\s+"); - status = qstatLineParts[QueueCommandManager.STATUS_INDEX]; - } - - public long getId() { - return id; - } - - public String getName() { - return name; - } - - public String getStatus() { - return status; - } - - @Override - public int hashCode() { - return Objects.hash(id); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - QstatEntry other = (QstatEntry) obj; - if (id != other.id) { - return false; - } - return true; - } - - @Override - public String toString() { - return id + ""; - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/remote/QstatParser.java b/src/main/java/gov/nasa/ziggy/module/remote/QstatParser.java new file mode 100644 index 0000000..c28d9b0 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/remote/QstatParser.java @@ -0,0 +1,155 @@ +package gov.nasa.ziggy.module.remote; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.stream.Collectors; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; + +/** + * Provides parsing capabilities for the outputs from a qstat command. + * + * @author PT + */ +public class QstatParser { + + private String owner; + private String serverName; + private QueueCommandManager cmdManager; + + /** Constructor for general use. */ + public QstatParser() { + this(QueueCommandManager.newInstance()); + } + + /** + * Constructor for test. + */ + QstatParser(QueueCommandManager cmdManager) { + this(cmdManager.user(), cmdManager.hostname(), cmdManager); + } + + private QstatParser(String owner, String serverName, QueueCommandManager cmdManager) { + this.owner = owner; + this.serverName = serverName; + if (cmdManager == null) { + this.cmdManager = QueueCommandManager.newInstance(); + } else { + this.cmdManager = cmdManager; + } + } + + /** Populates the Job IDs in a collection of {@link RemoteJobInformation} instances. */ + public void populateJobIds(PipelineTask pipelineTask, + List remoteJobsInformation) { + Map jobIdByName = jobIdByName(pipelineTask.taskBaseName()); + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + remoteJobInformation.setJobId(jobIdByName.get(remoteJobInformation.getJobName())); + } + } + + public RemoteJobInformation remoteJobInformation(RemoteJob remoteJob) { + return cmdManager.remoteJobInformation(remoteJob); + } + + /** + * Uses the output from qstat to generate a {@link Map} from the job name to the job ID. The + * process eliminates any jobs that have the wrong username or hostname, since those are + * obviously somebody else's jobs. + */ + public Map jobIdByName(String taskName) { + + Map jobIdByName = new HashMap<>(); + + // get all the qstat entries, if any, for the specified task name + List qstatLines = cmdManager.getQstatInfoByTaskName(owner, taskName); + if (qstatLines.isEmpty() || qstatLines.size() == 1 && qstatLines.get(0).isBlank()) { + return jobIdByName; + } + + // generate a QstatEntry for each string + List qstatEntries = qstatLines.stream() + .map(QstatEntry::new) + .collect(Collectors.toList()); + + // get the server names for all the jobs + Map jobIdToServerMap = cmdManager.serverNames( + qstatEntries.stream().map(QstatEntry::getId).sorted().collect(Collectors.toList())); + + // strip out the jobs with the wrong server names + List jobsWithCorrectServer = jobIdToServerMap.keySet() + .stream() + .filter(s -> jobIdToServerMap.get(s).equals(serverName)) + .collect(Collectors.toList()); + + // if the list is empty, return a null + if (jobsWithCorrectServer.isEmpty()) { + return jobIdByName; + } + + qstatEntries = qstatEntries.stream() + .filter(s -> jobsWithCorrectServer.contains(s.getId())) + .collect(Collectors.toList()); + + for (QstatEntry qstatEntry : qstatEntries) { + jobIdByName.put(qstatEntry.name, qstatEntry.id); + } + return jobIdByName; + } + + private static class QstatEntry { + + private final long id; + private final String name; + + /** + * Constructs a QstatEntry instance from a line out of a qstat command. + * + * @param qstatLine Line from the output of the qstat command + */ + public QstatEntry(String qstatLine) { + + String[] qstatLineParts = qstatLine.split("\\s+"); + + // The job ID has an ID # followed by ".". + String jobId = qstatLineParts[QueueCommandManager.ID_INDEX]; + String[] jobIdParts = jobId.split("\\."); + id = Long.parseLong(jobIdParts[0]); + + // The name, owner, and status are simple strings. + name = qstatLineParts[QueueCommandManager.NAME_INDEX]; + } + + public long getId() { + return id; + } + + @Override + public int hashCode() { + return Objects.hash(id); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + QstatEntry other = (QstatEntry) obj; + if (id != other.id) { + return false; + } + return true; + } + + @Override + public String toString() { + return id + ""; + } + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java b/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java index 46ff637..d2ecdd2 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java @@ -5,7 +5,9 @@ import java.io.UncheckedIOException; import java.util.ArrayList; import java.util.Date; +import java.util.HashMap; import java.util.List; +import java.util.Map; import org.apache.commons.configuration2.ImmutableConfiguration; import org.apache.commons.exec.CommandLine; @@ -64,15 +66,23 @@ public class Qsub { private Qsub() { } - public int[] submitMultipleJobsForTask() { - int[] returnCodes = new int[numNodes]; + /** + * Generates and executes the PBS commands that insert jobs into the job queue. + * + * @return A {@link Map} in which the keys are {@link RemoteJobInformation} instances, one for + * each submitted job, and the values are the return codes from the PBS commands. + */ + public Map submitJobsForTask() { + Map pbsStatusByJobInformation = new HashMap<>(); for (int iJob = 0; iJob < numNodes; iJob++) { - returnCodes[iJob] = submit1Job(iJob); + RemoteJobInformation remoteJobInformation = new RemoteJobInformation(pbsLogFile(iJob), + fullJobName(iJob)); + pbsStatusByJobInformation.put(remoteJobInformation, submitJobForTask(iJob)); } - return returnCodes; + return pbsStatusByJobInformation; } - private int submit1Job(int jobIndex) { + private int submitJobForTask(int jobIndex) { CommandLine commandLine = new CommandLine("/PBS/bin/qsub"); commandLine.addArgument("-N"); @@ -101,12 +111,12 @@ private int submit1Job(int jobIndex) { commandLine.addArgument(ziggyProgram); commandLine.addArgument(taskDir); - log.info("commandLine: " + commandLine); + log.info("commandLine={}", commandLine); for (String arg : commandLine.getArguments()) { - log.info("arg: " + arg); + log.info("arg={}", arg); } - log.info("Running PBS command: " + commandLine); + log.info("Running PBS command: {}", commandLine); return runCommandLine(commandLine, QSUB_TIMEOUT_SECS); } @@ -146,16 +156,6 @@ private List systemProperties(int jobIndex) { systemProperties.add(TaskLog.algorithmNameSystemProperty(pipelineTask)); return systemProperties; } -// @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) -// private String algorithmLogFile(int jobIndex) { -// File nasLogFile = new File(nasLogDir, pipelineTask.logFilename(jobIndex)); -// try { -// return nasLogFile.getCanonicalPath(); -// } catch (IOException e) { -// throw new UncheckedIOException("Unable to get path to file " + nasLogFile.toString(), -// e); -// } -// } private String resourceOptions(boolean multiNodeJob) { int numNodesInCall = 1; @@ -224,8 +224,8 @@ private void validate() { unsetParameters.add("pipelineTask"); } if (!unsetParameters.isEmpty()) { - log.error("Qsub missing required parameters: [" - + StringUtils.join(unsetParameters.iterator(), ",") + "]"); + log.error("Qsub missing required parameters [{}]", + StringUtils.join(unsetParameters.iterator(), ",")); throw new IllegalStateException("Qsub missing required parameters"); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/QsubLog.java b/src/main/java/gov/nasa/ziggy/module/remote/QsubLog.java index 33dd9e7..3261ac6 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/QsubLog.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/QsubLog.java @@ -42,8 +42,8 @@ public QsubLog(File logDirectory) { @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public void log(String qsubCommandLine) { - try (BufferedWriter bw = new BufferedWriter( - new OutputStreamWriter(new FileOutputStream(logFile, true), ZiggyFileUtils.ZIGGY_CHARSET))) { + try (BufferedWriter bw = new BufferedWriter(new OutputStreamWriter( + new FileOutputStream(logFile, true), ZiggyFileUtils.ZIGGY_CHARSET))) { // append to existing file if it exists bw.write(qsubCommandLine + "\n"); } catch (IOException e) { diff --git a/src/main/java/gov/nasa/ziggy/module/remote/QueueCommandManager.java b/src/main/java/gov/nasa/ziggy/module/remote/QueueCommandManager.java index bff4c76..fe3e9d3 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/QueueCommandManager.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/QueueCommandManager.java @@ -3,22 +3,17 @@ import java.lang.reflect.InvocationTargetException; import java.util.Collection; import java.util.HashMap; -import java.util.HashSet; import java.util.List; import java.util.Map; -import java.util.Set; import java.util.regex.Matcher; import java.util.regex.Pattern; -import org.apache.commons.collections.CollectionUtils; import org.apache.commons.configuration2.ImmutableConfiguration; import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.module.StateFile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.RemoteJob; import gov.nasa.ziggy.pipeline.definition.RemoteJob.RemoteJobQstatInfo; import gov.nasa.ziggy.services.config.PropertyName; @@ -29,11 +24,11 @@ /** * Executes batch commands and parses the text output from those commands. *

- * Subclasses of the {@link QueueCommandManager} class support the {@link QstatMonitor} class by + * Subclasses of the {@link QueueCommandManager} class support the {@link QstatParser} class by * obtaining job information that is only available in text output from the qstat shell command. The * command is executed, its text results captured, and the text is parsed to obtain the desired - * information for {@link QstatMonitor}. Information such as job IDs, job exit status, etc., can - * thus be obtained. + * information for {@link QstatParser}. Information such as job IDs, job exit status, etc., can thus + * be obtained. *

* Additionally, class instances can execute the qdel shell command to delete queued or running jobs * from the batch system. This allows the operator to terminate jobs via the pipeline user interface @@ -63,13 +58,17 @@ public abstract class QueueCommandManager { private static final Pattern EXIT_STATUS_PATTERN = Pattern .compile("\\s*Exit_status = (-?[0-9]+)"); - private static final Pattern EXIT_COMMENT_PATTERN = Pattern.compile("\\s*comment = (.+)"); public static final String DEFAULT_LOCAL_CLASS = "gov.nasa.ziggy.module.remote.QueueLocalCommandManager"; public static final String SELECT = "Resource_List.select"; public static final String WALLTIME = "resources_used.walltime"; + public static final String JOBNAME = "Job_Name"; + public static final String OUTPUT_PATH = "Output_Path"; public static final Pattern SELECT_PATTERN = Pattern .compile("\\s*" + SELECT + " = ([0-9]+):model=(\\S+)"); public static final Pattern WALLTIME_PATTERN = Pattern.compile("\\s*" + WALLTIME + " = (\\S+)"); + public static final Pattern JOBNAME_PATTERN = Pattern.compile("\\s*" + JOBNAME + " = (\\S+)"); + public static final Pattern OUTPUT_PATH_PATTERN = Pattern + .compile("\\s*" + OUTPUT_PATH + " = [^:]+:(\\S+)"); @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public static QueueCommandManager newInstance() { @@ -144,32 +143,6 @@ public Map serverNames(Collection jobIds) { return jobIdServerNameMap; } - /** - * Gets the qstat output line for a collection of jobs using their job IDs. - * - * @param jobIds IDs of jobs. - * @return Map from the job names (not IDs) to the qstat line for each job. - */ - public Map getQstatInfoByJobNameMap(Collection jobIds) { - - Map qstatInfoByJobName = new HashMap<>(); - - // construct the qstat arguments - String qstatArgs = qstatArgsWithJobIds("-x", jobIds); - - // run the command - List qstatOutput = qstat(qstatArgs); - - // skip the first 3 lines of the output and then parse 1 line per job - for (int i = 3; i < qstatOutput.size(); i++) { - String line = qstatOutput.get(i); - String jobName = line.split("\\s+")[NAME_INDEX]; - qstatInfoByJobName.put(jobName, line); - } - - return qstatInfoByJobName; - } - /** * Gets the exit status for a job. * @@ -198,7 +171,7 @@ public RemoteJob.RemoteJobQstatInfo remoteJobQstatInfo(long jobId) { RemoteJobQstatInfo remoteJobQstatInfo = new RemoteJobQstatInfo(); String qstatArgs = "-xf " + jobId; List qstatOutput = qstat(qstatArgs, SELECT, WALLTIME); - log.debug("job " + jobId + "qstat lines: " + qstatOutput.toString()); + log.debug("job {}, qstat lines {}", jobId, qstatOutput.toString()); String selectLine = null; String walltimeLine = null; @@ -227,6 +200,22 @@ public RemoteJob.RemoteJobQstatInfo remoteJobQstatInfo(long jobId) { return remoteJobQstatInfo; } + public RemoteJobInformation remoteJobInformation(RemoteJob remoteJob) { + String qstatArgs = "-xf " + remoteJob.getJobId(); + List qstatOutput = qstat(qstatArgs, JOBNAME, OUTPUT_PATH); + log.debug("job {}, qstat lines {}", remoteJob.getJobId(), qstatOutput.toString()); + Matcher m = JOBNAME_PATTERN.matcher(qstatOutput.get(0)); + String jobName = m.matches() ? m.group(1) : null; + m = OUTPUT_PATH_PATTERN.matcher(qstatOutput.get(1)); + String logFile = m.matches() ? m.group(1) : null; + if (StringUtils.isBlank(jobName) || StringUtils.isBlank(logFile)) { + return null; + } + RemoteJobInformation remoteJobInformation = new RemoteJobInformation(logFile, jobName); + remoteJobInformation.setJobId(remoteJob.getJobId()); + return remoteJobInformation; + } + /** * Matches a string to a pattern and returns a single specified group. * @@ -242,82 +231,11 @@ private String matchAndReturn(Pattern pattern, String qstatOutputString, int gro return null; } - /** - * Gets the exit comment for a job. - * - * @param jobId ID number for the job - * @return exit comment, can be null if no job with the specified ID can be found - */ - public String exitComment(long jobId) { - String qstatArgs = "-xf " + jobId; - List qstatOutput = qstat(qstatArgs, "comment"); - if (qstatOutput != null && !qstatOutput.isEmpty()) { - String exitCommentString = qstatOutput.get(0); - if (!StringUtils.isBlank(exitCommentString)) { - return matchAndReturn(EXIT_COMMENT_PATTERN, exitCommentString, 1); - } - } - return null; - } - - /** - * Issues the delete command for all selected pipeline tasks. - * - * @param pipelineTasks selected pipeline tasks. - */ - public int deleteJobsForPipelineTasks(List pipelineTasks) { - String qdelArgs = qdelArgsForPipelineTasks(pipelineTasks); - return qdel(qdelArgs); - } - public int deleteJobsByJobId(Collection jobIds) { String qdelArgs = qdelArgsForJobIds(jobIds); return qdel(qdelArgs); } - /** Issues the delete command for a single task, based on its state file. */ - public int deleteJobsForStateFile(StateFile stateFile) { - - // Add the task to monitoring. - QstatMonitor monitor = new QstatMonitor(this); - monitor.addToMonitoring(stateFile); - monitor.update(); - - // Get the job IDs. - Set jobIds = monitor.allIncompleteJobIds(stateFile); - if (!CollectionUtils.isEmpty(jobIds)) { - return qdel(qdelArgsForJobIds(jobIds)); - } - return 0; - } - - /** - * Generates the qdel command string. - * - * @param pipelineTasks pipeline tasks selected for deletion. - * @return Command string for deletion using qdel. - */ - private String qdelArgsForPipelineTasks(List pipelineTasks) { - - // obtain the job IDs for each task. This is accomplished via the - // QstatMonitor - QstatMonitor monitor = new QstatMonitor(this); - for (PipelineTask task : pipelineTasks) { - monitor.addToMonitoring(task); - } - monitor.update(); - - // Get the job IDs. - Set allJobIds = new HashSet<>(); - for (PipelineTask task : pipelineTasks) { - Set jobIds = monitor.allIncompleteJobIds(task); - if (jobIds != null) { - allJobIds.addAll(jobIds); - } - } - return qdelArgsForJobIds(allJobIds); - } - private String qdelArgsForJobIds(Collection jobIds) { if (jobIds == null || jobIds.isEmpty()) { return null; @@ -329,10 +247,6 @@ private String qdelArgsForJobIds(Collection jobIds) { return qdelCommand.toString(); } - private List qstat(String commandOptions) { - return qstat(commandOptions, (String[]) null); - } - /** * Executes the qstat command string and returns the result as a List<String>, one list * entry per line, filtering to return only those lines that contain one or more target strings. diff --git a/src/main/java/gov/nasa/ziggy/module/remote/QueueLocalCommandManager.java b/src/main/java/gov/nasa/ziggy/module/remote/QueueLocalCommandManager.java index 42e57bc..9f37cee 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/QueueLocalCommandManager.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/QueueLocalCommandManager.java @@ -50,7 +50,7 @@ protected List qstat(String commandArgs, String... targets) { @Override protected int qdel(String qdelArgs) { ExternalProcess p = ExternalProcess.simpleExternalProcess(QDEL + qdelArgs); - log.info("Executing command: " + qdelArgs); + log.info("Executing command {}", qdelArgs); return p.run(false, 0); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/RemoteExecutor.java b/src/main/java/gov/nasa/ziggy/module/remote/RemoteExecutor.java index bad0ca0..9bcdc81 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/RemoteExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/RemoteExecutor.java @@ -4,25 +4,40 @@ import java.io.IOException; import java.io.UncheckedIOException; import java.nio.file.Path; - +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; +import java.util.stream.Collectors; + +import org.apache.commons.collections.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.AlgorithmExecutor; -import gov.nasa.ziggy.module.StateFile; +import gov.nasa.ziggy.module.AlgorithmMonitor; +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.messages.MonitorAlgorithmRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.TimeFormatter; public abstract class RemoteExecutor extends AlgorithmExecutor { private static final Logger log = LoggerFactory.getLogger(RemoteExecutor.class); + private List remoteJobsInformation; + private QstatParser qstatParser = new QstatParser(); + protected RemoteExecutor(PipelineTask pipelineTask) { super(pipelineTask); } @@ -32,83 +47,136 @@ protected RemoteExecutor(PipelineTask pipelineTask) { * remote clusters. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - protected void submitToPbsInternal(StateFile initialState, PipelineTask pipelineTask, - Path algorithmLogDir, Path taskDir) { + protected void submitToPbsInternal(PipelineTask pipelineTask) { try { - getStateFile().setPbsSubmitTimeMillis(System.currentTimeMillis()); - getStateFile().persist(); - addToMonitor(getStateFile()); - - log.info("Launching jobs for state file: " + getStateFile()); - - int coresPerNode = getStateFile().getActiveCoresPerNode(); + log.info("Launching jobs for task {}", pipelineTask.taskBaseName()); - int numNodes = getStateFile().getRequestedNodeCount(); - log.info("Number of nodes requested for jobs: " + numNodes); + int numNodes = getPbsParameters().getRequestedNodeCount(); + log.info("Requested {} nodes for jobs", numNodes); - log.info("Launching job, name=" + pipelineTask.taskBaseName() + ", coresPerNode=" - + coresPerNode + ", numNodes=" + numNodes); + log.info("Launching job, name={}, coresPerNode={}, numNodes={}", + pipelineTask.taskBaseName(), getPbsParameters().getActiveCoresPerNode(), numNodes); File pbsLogDirFile = DirectoryProperties.pbsLogDir().toFile(); pbsLogDirFile.mkdirs(); String pbsLogDir = pbsLogDirFile.getCanonicalPath(); - Qsub.Builder b = new Qsub.Builder().queueName(getStateFile().getQueueName()) - .wallTime(getStateFile().getRequestedWallTime()) + Qsub.Builder b = new Qsub.Builder().queueName(getPbsParameters().getQueueName()) + .wallTime(getPbsParameters().getRequestedWallTime()) .numNodes(numNodes) - .coresPerNode(getStateFile().getMinCoresPerNode()) - .gigsPerNode(getStateFile().getMinGigsPerNode()) - .model(getStateFile().getRemoteNodeArchitecture()) - .groupName(getStateFile().getRemoteGroup()) + .coresPerNode(getPbsParameters().getMinCoresPerNode()) + .gigsPerNode(getPbsParameters().getMinGigsPerNode()) + .model(getPbsParameters().getArchitecture().getNodeName()) + .groupName(getPbsParameters().getRemoteGroup()) .scriptPath(new File(DirectoryProperties.ziggyBinDir().toFile(), ZIGGY_PROGRAM) .getCanonicalPath()) .ziggyProgram(NODE_MASTER_NAME) .pbsLogDir(pbsLogDir) .pipelineTask(pipelineTask) .taskDir(workingDir().toString()); - b.cluster(RemoteQueueDescriptor.fromQueueName(getStateFile().getQueueName()) + b.cluster(RemoteQueueDescriptor.fromQueueName(getPbsParameters().getQueueName()) .getRemoteCluster()); Qsub qsub = b.build(); log.info("Submitting multiple jobs with 1 node per job for this task"); - int[] returnCodes = qsub.submitMultipleJobsForTask(); - int goodJobsCount = 0; - for (int returnCode : returnCodes) { - if (returnCode == 0) { - goodJobsCount++; - } + Map jobStatusByJobInformation = qsub.submitJobsForTask(); + int attemptedJobSubmissions = jobStatusByJobInformation.size(); + List successfullySubmittedJobsInformation = jobStatusByJobInformation + .entrySet() + .stream() + .filter(s -> s.getValue().intValue() == 0) + .map(Entry::getKey) + .collect(Collectors.toList()); + if (successfullySubmittedJobsInformation.size() < jobStatusByJobInformation.size()) { + log.warn("{} jobs submitted but only {} successfully queued", + attemptedJobSubmissions, successfullySubmittedJobsInformation.size()); + AlertService.getInstance() + .generateAndBroadcastAlert("PI (Remote)", pipelineTask, Severity.WARNING, + "Attempted to submit " + attemptedJobSubmissions + + " jobs but only successfully submitted " + + successfullySubmittedJobsInformation.size()); } - if (goodJobsCount < returnCodes.length) { - log.warn(returnCodes.length + " jobs submitted but only " + goodJobsCount - + " successfully queued"); + if (successfullySubmittedJobsInformation.size() == 0) { AlertService.getInstance() - .generateAndBroadcastAlert("PI (Remote)", getStateFile().getPipelineTaskId(), - AlertService.Severity.WARNING, "Attempted to submit " + returnCodes.length - + " jobs but only successfully submitted " + goodJobsCount); + .generateAndBroadcastAlert("PI (Remote)", pipelineTask, Severity.ERROR, + "No remote jobs submitted"); + + throw new PipelineException("No remote jobs created for task " + pipelineTask); } + remoteJobsInformation = successfullySubmittedJobsInformation; - // Update the task log index. - pipelineTaskOperations().setRemoteExecution(pipelineTask.getId()); + // Add the job IDs to the RemoteJobInformation instances. + qstatParser().populateJobIds(pipelineTask, successfullySubmittedJobsInformation); - // Update the remote jobs in the database - pipelineTaskOperations().createRemoteJobsFromQstat(pipelineTask.getId()); + // Update the remote jobs in the database. + pipelineTaskDataOperations().addRemoteJobs(pipelineTask, + successfullySubmittedJobsInformation); - log.info("Updating processing step -> " + ProcessingStep.QUEUED); + addToMonitor(); - pipelineTaskOperations().updateProcessingStep(pipelineTask.getId(), - ProcessingStep.QUEUED); - pipelineTaskOperations().updateSubtaskCounts(pipelineTask.getId(), - initialState.getNumTotal(), initialState.getNumComplete(), - initialState.getNumFailed()); + // Update the task log index. + pipelineTaskDataOperations().updateAlgorithmType(pipelineTask, AlgorithmType.REMOTE); + + log.info("Updating processing step -> {}", ProcessingStep.QUEUED); + + pipelineTaskDataOperations().updateSubtaskCounts(pipelineTask, -1, 0, 0); + pipelineTaskDataOperations().updateProcessingStep(pipelineTask, ProcessingStep.QUEUED); } catch (IOException e) { throw new UncheckedIOException("Submit to PBS failed due to IOException", e); } } + /** + * Resumes monitoring of the task in the {@link AlgorithmMonitor}. + *

+ * The {@link RemoteJob} instances for the given {@link PipelineTask} are retrieved and new + * {@link RemoteJobInformation} instances are created for any remote jobs that are incomplete. + * If any RemoteJobInformation instances are created, the task is sent to the AlgorithmMonitor. + * If no RemoteJobInformation instances are created, it indicates that there are no incomplete + * remote jobs for the task, hence no monitoring resumption is attempted. + * + * @return true if there were jobs to resume, false otherwise + */ + public boolean resumeMonitoring() { + remoteJobsInformation = new ArrayList<>(); + Set remoteJobs = pipelineTaskDataOperations().remoteJobs(pipelineTask); + for (RemoteJob remoteJob : remoteJobs) { + if (remoteJob.isFinished()) { + continue; + } + RemoteJobInformation remoteJobInformation = qstatParser() + .remoteJobInformation(remoteJob); + if (remoteJobInformation == null) { + continue; + } + remoteJobsInformation.add(remoteJobInformation); + } + if (CollectionUtils.isEmpty(remoteJobsInformation)) { + log.info("No running remote jobs detected, monitoring will not resume"); + return false; + } + log.info("{} running remote jobs detected, monitoring will resume", + remoteJobsInformation.size()); + addToMonitor(); + return true; + } + + @Override + protected void addToMonitor() { + ZiggyMessenger.publish( + new MonitorAlgorithmRequest(pipelineTask, workingDir(), remoteJobsInformation)); + } + @Override - protected void addToMonitor(StateFile stateFile) { - ZiggyMessenger.publish(new MonitorAlgorithmRequest(stateFile, algorithmType())); + public String activeCores() { + return Integer.toString(getPbsParameters().getActiveCoresPerNode()); + } + + @Override + public String wallTime() { + return Integer.toString((int) TimeFormatter + .timeStringHhMmSsToTimeInSeconds(getPbsParameters().getRequestedWallTime())); } @Override @@ -125,4 +193,12 @@ protected Path taskDataDir() { public AlgorithmType algorithmType() { return AlgorithmType.REMOTE; } + + QstatParser qstatParser() { + return qstatParser; + } + + List getRemoteJobsInformation() { + return remoteJobsInformation; + } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/RemoteJobInformation.java b/src/main/java/gov/nasa/ziggy/module/remote/RemoteJobInformation.java new file mode 100644 index 0000000..1216a10 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/remote/RemoteJobInformation.java @@ -0,0 +1,63 @@ +package gov.nasa.ziggy.module.remote; + +import java.io.Serializable; +import java.util.Objects; + +/** + * Container class for information about a remote job. + *

+ * Note that the job ID cannot be made final, nor assigned at create time. This is because the job + * IDs won't be known until later, hence will need to be assigned when they become known. + * + * @author PT + */ +public class RemoteJobInformation implements Serializable { + + private static final long serialVersionUID = 20240822L; + + private final String logFile; + private final String jobName; + private long jobId; + + public RemoteJobInformation(String logFile, String jobName) { + this.logFile = logFile; + this.jobName = jobName; + } + + public String getLogFile() { + return logFile; + } + + public String getJobName() { + return jobName; + } + + public void setJobId(long jobId) { + this.jobId = jobId; + } + + public long getJobId() { + return jobId; + } + + @Override + public int hashCode() { + return Objects.hash(jobId, jobName, logFile); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (getClass() != obj.getClass()) { + return false; + } + RemoteJobInformation other = (RemoteJobInformation) obj; + return jobId == other.jobId && Objects.equals(jobName, other.jobName) + && Objects.equals(logFile, other.logFile); + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java b/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java index 1ccde3f..559f8e5 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.module.remote.aws; -import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteExecutor; import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; @@ -73,7 +72,7 @@ public PbsParameters generatePbsParameters( } @Override - protected void submitForExecution(StateFile stateFile) { - submitToPbsInternal(stateFile, pipelineTask, algorithmLogDir(), taskDataDir()); + protected void submitForExecution() { + submitToPbsInternal(pipelineTask); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java index 1d913ee..cbf4004 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.module.remote.nas; -import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteExecutor; import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; @@ -41,7 +40,7 @@ public PbsParameters generatePbsParameters( } @Override - protected void submitForExecution(StateFile stateFile) { - submitToPbsInternal(stateFile, pipelineTask, algorithmLogDir(), taskDataDir()); + protected void submitForExecution() { + submitToPbsInternal(pipelineTask); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java index dfb311f..1bcb77c 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java @@ -137,7 +137,7 @@ public void populate(String qsResultsDestination) { Set allMetrics = parseQsCsv(qsResultsDestination, directorate); for (ArchitectureQueueTimeMetrics metrics : allMetrics) { - log.info("Architecture string: " + metrics.archName); + log.info("Architecture string {}", metrics.archName); RemoteNodeDescriptor descriptor = remoteNodeNameMap.get(metrics.archName); if (descriptor != null) { queueDepthMap.put(descriptor, metrics.runoutTime); @@ -165,8 +165,8 @@ private Set parseQsCsv(String file, String nasaDiv CSVFormat format = CSVFormat.Builder.create(CSVFormat.EXCEL) .setRecordSeparator("\n") .build(); - try (CSVParser parser = format.parse( - new InputStreamReader(new FileInputStream(new File(file)), ZiggyFileUtils.ZIGGY_CHARSET))) { + try (CSVParser parser = format.parse(new InputStreamReader( + new FileInputStream(new File(file)), ZiggyFileUtils.ZIGGY_CHARSET))) { List csvRecords = parser.getRecords(); CSVRecord divisionsRecord = csvRecords.get(0); List divisions = new ArrayList<>(); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java index 0a0f6a7..b817305 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java @@ -25,18 +25,20 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.database.RuntimeObjectFactory; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.events.ZiggyEventHandler; import gov.nasa.ziggy.services.messages.RemoveTaskFromKilledTasksMessage; import gov.nasa.ziggy.services.messages.TaskRequest; @@ -65,6 +67,8 @@ public class PipelineExecutor { private static Map> instanceEventLabels = new HashMap<>(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); private PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); private ModelOperations modelOperations = new ModelOperations(); @@ -180,7 +184,7 @@ private void restartFailedTasksForNode( log.info("Restarting {} tasks for instance node {} ({})", tasks.size(), node.getId(), node.getModuleName()); - pipelineInstanceNodeOperations().taskCounts(node); + pipelineTaskDisplayDataOperations().taskCounts(node); logInstanceNodeCounts(node, "initial"); // Loop over tasks and prepare for restart, including sending the task request message. @@ -195,7 +199,7 @@ private void restartFailedTasksForNode( } private void logInstanceNodeCounts(PipelineInstanceNode node, String initialOrFinal) { - TaskCounts instanceNodeCounts = pipelineInstanceNodeOperations().taskCounts(node); + TaskCounts instanceNodeCounts = pipelineTaskDisplayDataOperations().taskCounts(node); log.info("node={}: {}", initialOrFinal, instanceNodeCounts); } @@ -204,37 +208,20 @@ private void logInstanceNodeCounts(PipelineInstanceNode node, String initialOrFi */ private void restartFailedTask(PipelineTask task, boolean doTransitionOnly, RunMode restartMode) { - boolean okayToRestart = false; - if (task.isError() || task.getProcessingStep() == ProcessingStep.COMPLETE - && task.getFailedSubtaskCount() > 0) { - okayToRestart = true; - } - ProcessingStep oldProcessingStep = task.getProcessingStep(); - if (!okayToRestart) { - log.warn("Task {} is on step {} {} errors, not restarting", task.getId(), - oldProcessingStep, task.isError() ? "with" : "without"); - return; + pipelineTaskDataOperations().prepareTaskForRestart(task); + removeTaskFromKilledTaskList(task); + if (restartMode != RunMode.RESUME_CURRENT_STEP) { + pipelineTaskDataOperations().updateProcessingStep(task, ProcessingStep.WAITING_TO_RUN); } - // Retrieve the task so that it can be modified in the database using the Hibernate - // infrastructure - PipelineTask databaseTask = pipelineTaskOperations().pipelineTask(task.getId()); - log.info("Restarting task {} on step {} {} errors", databaseTask.getId(), oldProcessingStep, - task.isError() ? "with" : "without"); - - PipelineTask mergedTask = pipelineTaskOperations().clearError(task.getId()); - mergedTask = pipelineTaskOperations().updateProcessingStep(mergedTask, - ProcessingStep.WAITING_TO_RUN); - removeTaskFromKilledTaskList(mergedTask.getId()); - // Send the task message to the supervisor. - sendTaskRequestMessage(mergedTask, Priority.HIGHEST, doTransitionOnly, restartMode); + sendTaskRequestMessage(task, Priority.HIGHEST, doTransitionOnly, restartMode); } /** Replace with dummy method in unit testing. */ - public void removeTaskFromKilledTaskList(long taskId) { - ZiggyMessenger.publish(new RemoveTaskFromKilledTasksMessage(taskId)); + public void removeTaskFromKilledTaskList(PipelineTask pipelineTask) { + ZiggyMessenger.publish(new RemoveTaskFromKilledTasksMessage(pipelineTask)); } /** @@ -262,12 +249,12 @@ private void launchNode(PipelineInstanceNode instanceNode) { List unitsOfWork = new ArrayList<>(); unitsOfWork = generateUnitsOfWork(unitOfWorkGenerator.newInstance(), instanceNode, instance); - log.info("Generated " + unitsOfWork.size() + " tasks for pipeline definition node " - + instanceNode.getModuleName()); + log.info("Generated {} tasks for pipeline definition node {}", unitsOfWork.size(), + instanceNode.getModuleName()); if (unitsOfWork.isEmpty()) { AlertService.getInstance() - .generateAndBroadcastAlert("PI", AlertService.DEFAULT_TASK_ID, Severity.ERROR, + .generateAndBroadcastAlert("PI", AlertService.DEFAULT_TASK, Severity.ERROR, "No tasks generated for " + instanceNode.getModuleName()); pipelineInstanceOperations().setInstanceToErrorsStalledState(instance); throw new PipelineException("Task generation did not generate any tasks! UOW class: " @@ -305,12 +292,12 @@ private void sendTaskRequestMessage(PipelineTask task, Priority priority, return; } - log.debug("Generating worker task message for task=" + task.getId() + ", module=" - + task.getModuleName()); + log.debug("Generating worker task message for task {}, module {}", task, + task.getModuleName()); TaskRequest taskRequest = new TaskRequest(task.getPipelineInstanceId(), pipelineTaskOperations().pipelineInstanceNodeId(task), - pipelineTaskOperations().pipelineDefinitionNode(task).getId(), task.getId(), priority, + pipelineTaskOperations().pipelineDefinitionNode(task).getId(), task, priority, doTransitionOnly, runMode); ZiggyMessenger.publish(taskRequest); @@ -329,7 +316,7 @@ public static List generateUnitsOfWork(UnitOfWorkGenerator uowGenera */ public static List generateUnitsOfWork(UnitOfWorkGenerator uowGenerator, PipelineInstanceNode pipelineInstanceNode, PipelineInstance instance) { - log.debug("Generating UOWs..."); + log.debug("Generating UOWs"); // Produce the tasks. Set eventLabels = instance != null ? instanceEventLabels.get(instance.getId()) : null; @@ -351,6 +338,14 @@ public PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + ModelOperations modelOperations() { return modelOperations; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineReportGenerator.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineReportGenerator.java index e96297b..475126e 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineReportGenerator.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineReportGenerator.java @@ -18,6 +18,8 @@ import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -35,6 +37,8 @@ public class PipelineReportGenerator { private ParametersOperations parametersOperations = new ParametersOperations(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); private PipelineDefinitionNodeOperations pipelineDefinitionNodeOperations = new PipelineDefinitionNodeOperations(); private PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations = new PipelineModuleDefinitionOperations(); private PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); @@ -102,7 +106,7 @@ public String generatePedigreeReport(PipelineInstance instance) { report.append("Instance Name: " + instance.getName() + nl); report.append("Instance Priority: " + instance.getPriority() + nl); report.append("Instance State: " + instance.getState() + nl); - List instanceSoftwareRevisions = pipelineInstanceOperations() + List instanceSoftwareRevisions = pipelineTaskDataOperations() .distinctSoftwareRevisions(instance); report.append("Instance Software Revisions: " + instanceSoftwareRevisions + nl); report.append(nl); @@ -126,7 +130,7 @@ public String generatePedigreeReport(PipelineInstance instance) { for (PipelineInstanceNode node : pipelineNodes) { PipelineModuleDefinition module = node.getPipelineModuleDefinition(); - TaskCounts instanceNodeCounts = pipelineInstanceNodeOperations().taskCounts(node); + TaskCounts instanceNodeCounts = pipelineTaskDisplayDataOperations().taskCounts(node); appendModule(nl, report, module); @@ -134,7 +138,7 @@ public String generatePedigreeReport(PipelineInstance instance) { + instanceNodeCounts.getTaskCount() + "/" + instanceNodeCounts.getTotalCounts().getCompletedTaskCount() + "/" + instanceNodeCounts.getTotalCounts().getFailedTaskCount() + nl); - List nodeSoftwareRevisions = pipelineInstanceNodeOperations() + List nodeSoftwareRevisions = pipelineTaskDataOperations() .distinctSoftwareRevisions(node); report.append( FOUR_SPACE_INDENT + "Software Revisions for node:" + nodeSoftwareRevisions + nl); @@ -221,6 +225,14 @@ PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + PipelineInstanceOperations pipelineInstanceOperations() { return pipelineInstanceOperations; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java index 8a013a5..74ac762 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java @@ -100,7 +100,7 @@ public static synchronized List subtaskInformation( */ private static synchronized void generateSubtaskInformation(PipelineDefinitionNode node) { - log.debug("Generating subtask information for node " + node.getModuleName()); + log.debug("Generating subtask information for node {}", node.getModuleName()); PipelineDefinition pipelineDefinition = instancePipelineDefinitionOperations() .pipelineDefinition(node.getPipelineName()); @@ -123,14 +123,14 @@ private static synchronized void generateSubtaskInformation(PipelineDefinitionNo ClassWrapper unitOfWorkGenerator = instance.unitOfWorkGenerator(node); // Generate the units of work. - List tasks = instance.unitsOfWork(unitOfWorkGenerator, instanceNode, + List unitsOfWork = instance.unitsOfWork(unitOfWorkGenerator, instanceNode, pipelineInstance); // Generate the subtask information for all tasks List subtaskInformationList = new LinkedList<>(); - for (UnitOfWork task : tasks) { - PipelineTask pipelineTask = instance.pipelineTask(pipelineInstance, instanceNode, task); - pipelineTask.setUowTaskParameters(task.getParameters()); + for (UnitOfWork unitOfWork : unitsOfWork) { + PipelineTask pipelineTask = instance.pipelineTask(pipelineInstance, instanceNode, + unitOfWork); SubtaskInformation subtaskInformation = instance.subtaskInformation(moduleDefinition, pipelineTask, node); subtaskInformationList.add(subtaskInformation); @@ -163,9 +163,7 @@ ClassWrapper unitOfWorkGenerator(PipelineDefinitionNode nod */ PipelineTask pipelineTask(PipelineInstance instance, PipelineInstanceNode instanceNode, UnitOfWork uow) { - PipelineTask pipelineTask = new PipelineTask(instance, instanceNode); - pipelineTask.setUowTaskParameters(uow.getParameters()); - return pipelineTask; + return new PipelineTask(instance, instanceNode, uow); } /** diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java index 95732ca..50f9f58 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java @@ -32,9 +32,7 @@ public class ClassWrapper implements Comparable> { } /** - * Construct with a new instance - * - * @param clazz + * Creates a {@code ClassWrapper} for the given type. */ public ClassWrapper(Class clazz) { this.clazz = clazz.getName(); @@ -42,11 +40,7 @@ public ClassWrapper(Class clazz) { } /** - * Construct from an existing instance - * - * @param - * @param instance - * @throws PipelineException + * Creates a {@code ClassWrapper} for the class of the given instance. */ public ClassWrapper(E instance) { clazz = instance.getClass().getName(); @@ -54,9 +48,7 @@ public ClassWrapper(E instance) { } /** - * Copy constructor - * - * @param otherClassWrapper + * Creates a copy of the given {@code ClassWrapper}. */ public ClassWrapper(ClassWrapper otherClassWrapper) { clazz = otherClassWrapper.clazz; @@ -79,7 +71,7 @@ public T newInstance() { } /** - * Returns a constructor with a particular signature + * Returns a constructor with a particular signature. */ @SuppressWarnings("unchecked") @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) @@ -111,9 +103,6 @@ public String unmangledClassName() { return unmangledClassName; } - /** - * @return the className - */ public String getClassName() { return clazz; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ExecutionClock.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ExecutionClock.java new file mode 100644 index 0000000..029a3e2 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ExecutionClock.java @@ -0,0 +1,116 @@ +package gov.nasa.ziggy.pipeline.definition; + +import java.util.Objects; + +import org.apache.commons.lang3.time.DurationFormatUtils; + +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.util.SystemProxy; +import jakarta.persistence.Embeddable; + +/** + * Provides total execution time for {@link PipelineTask} and {@link PipelineInstance} objects. All + * times are expressed in milliseconds since the start of the Unix epoch, with the exception of + * {@link #toString()}, which returns a {@link String} of the total execution time in "HH:mm:ss" + * format. + * + * @author PT + * @author Bill Wohler + */ +@Embeddable +public class ExecutionClock { + + /** Timestamp that processing most recently started. */ + private long startProcessingTime; + + /** Total time spent processing prior execution attempts. */ + private long priorProcessingExecutionTime; + + private boolean running; + + /** + * Starts execution timing for an instance. At start time, the processing-start time is set. + *

+ * Users are advised not to call this method directly. The {@link PipelineTaskOperations} class + * will call this method as needed. + */ + public void start() { + + // Only start the clock if it's not currently running. + if (running) { + return; + } + + startProcessingTime = SystemProxy.currentTimeMillis(); + running = true; + } + + /** + * Stops execution timing for an instance. The current-execution start time is set to -1 and the + * elapsed time from the current execution attempt is added to the total execution time. The end + * processing time is also set. + *

+ * Users are advised not to call this method directly. The {@link PipelineTaskOperations} class + * will call this method as needed. + */ + public void stop() { + + // Only stop the clock if it's currently running. + if (!running) { + return; + } + + priorProcessingExecutionTime += SystemProxy.currentTimeMillis() - startProcessingTime; + running = false; + } + + /** + * Calculates the total execution time for all processing attempts to date, in milliseconds. The + * total time includes the sum of the total execution time for all prior efforts; if a + * processing effort is current underway, the duration of the current effort is added to the + * total execution time from all prior efforts. + */ + public long totalExecutionTime() { + long totalExecutionTime = priorProcessingExecutionTime; + if (running) { + totalExecutionTime += SystemProxy.currentTimeMillis() - startProcessingTime; + } + return totalExecutionTime; + } + + public boolean isRunning() { + return running; + } + + @Override + public int hashCode() { + return Objects.hash(priorProcessingExecutionTime, running, startProcessingTime); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if ((obj == null) || (getClass() != obj.getClass())) { + return false; + } + ExecutionClock other = (ExecutionClock) obj; + return priorProcessingExecutionTime == other.priorProcessingExecutionTime + && running == other.running && startProcessingTime == other.startProcessingTime; + } + + /** + * Returns the elapsed time for processing (both current attempt and any prior ones) in + * "HH:mm:ss" format. If no processing has occurred, "-" is returned. This is the same time + * value that is returned by {@link #totalExecutionTime()}, but formatted for display + * convenience. + */ + @Override + public String toString() { + if (startProcessingTime == 0) { + return "-"; + } + return DurationFormatUtils.formatDuration(totalExecutionTime(), "HH:mm:ss"); + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java index 24866a6..f3be47e 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java @@ -14,26 +14,25 @@ import jakarta.persistence.ManyToOne; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; -import jakarta.persistence.Transient; import jakarta.persistence.UniqueConstraint; /** - * Group identifier for {@link PipelineDefinition}s, {@link PipelineModuleDefinition}s, and - * {@link ParameterSet}s. - *

- * Groups are used in the console to organize these entities into folders since their numbers can - * grow large over the course of the mission. + * Groups are used in the console to organize items into folders since their numbers can grow large + * over the course of the mission. *

* The names associated with a {@link Group} are represented as element collections with eager * fetching because we generally need all of the names whenever we access a group. * * @author Todd Klaus + * @author Bill Wohler */ @Entity -@Table(name = "ziggy_Group", uniqueConstraints = { @UniqueConstraint(columnNames = { "name" }) }) +@Table(name = "ziggy_Group", + uniqueConstraints = { @UniqueConstraint(columnNames = { "name", "type" }) }) public class Group { - public static final Group DEFAULT = new Group(); + public static final String DEFAULT_NAME = ""; + public static final Group DEFAULT = new Group(DEFAULT_NAME, null); @Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_Group_generator") @@ -43,38 +42,26 @@ public class Group { private String name; - @ManyToOne - private Group parentGroup; - - private String description; + private String type; - @ElementCollection(fetch = FetchType.EAGER) - @JoinTable(name = "ziggy_Group_pipelineDefinitionNames") - private Set pipelineDefinitionNames = new HashSet<>(); + @ManyToOne + private Group parent; @ElementCollection(fetch = FetchType.EAGER) - @JoinTable(name = "ziggy_Group_pipelineModuleNames") - private Set pipelineModuleNames = new HashSet<>(); + @JoinTable(name = "ziggy_Group_children") + private Set children = new HashSet<>(); @ElementCollection(fetch = FetchType.EAGER) - @JoinTable(name = "ziggy_Group_parameterSetNames") - private Set parameterSetNames = new HashSet<>(); - - /** Contains whichever of the above is correct for the current use-case. */ - @Transient - private Set memberNames; + @JoinTable(name = "ziggy_Group_items") + private Set items = new HashSet<>(); + // For Hibernate. Group() { } - public Group(String name) { + public Group(String name, String type) { this.name = name; - } - - public Group(Group group) { - name = group.name; - description = group.description; - parentGroup = group.parentGroup; + this.type = type; } public String getName() { @@ -88,62 +75,41 @@ public void setName(String name) { this.name = name; } - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - @Override - public String toString() { - return name; - } - - public Group getParentGroup() { - return parentGroup; - } - - public void setParentGroup(Group parentGroup) { - this.parentGroup = parentGroup; - } - - public Set getPipelineDefinitionNames() { - return pipelineDefinitionNames; + public String getType() { + return type; } - public void setPipelineDefinitionNames(Set pipelineDefinitionNames) { - this.pipelineDefinitionNames = pipelineDefinitionNames; + public void setType(String type) { + this.type = type; } - public Set getPipelineModuleNames() { - return pipelineModuleNames; + public Group getParent() { + return parent; } - public void setPipelineModuleNames(Set pipelineModuleNames) { - this.pipelineModuleNames = pipelineModuleNames; + public void setParent(Group parentGroup) { + parent = parentGroup; } - public Set getParameterSetNames() { - return parameterSetNames; + public Set getChildren() { + return children; } - public void setParameterSetNames(Set parameterSetNames) { - this.parameterSetNames = parameterSetNames; + public void setChildren(Set children) { + this.children = children; } - public Set getMemberNames() { - return memberNames; + public Set getItems() { + return items; } - public void setMemberNames(Set memberNames) { - this.memberNames = memberNames; + public void setItems(Set items) { + this.items = items; } @Override public int hashCode() { - return Objects.hash(name); + return Objects.hash(name, type); } @Override @@ -154,10 +120,12 @@ public boolean equals(Object obj) { if (obj == null || getClass() != obj.getClass()) { return false; } - final Group other = (Group) obj; - if (!Objects.equals(name, other.name)) { - return false; - } - return true; + Group other = (Group) obj; + return Objects.equals(name, other.name) && Objects.equals(type, other.type); + } + + @Override + public String toString() { + return name; } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java deleted file mode 100644 index d4b43f9..0000000 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java +++ /dev/null @@ -1,15 +0,0 @@ -package gov.nasa.ziggy.pipeline.definition; - -/** - * A database entity that can be assigned to a {@link Group}. - * - * @author PT - */ -public interface Groupable { - - /** - * Name of the object in the class that implements {@link Groupable}. Not to be confused with - * the name of the group itself. - */ - String getName(); -} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java index c706414..61c11ac 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java @@ -37,8 +37,7 @@ @Entity @Table(name = "ziggy_ParameterSet", uniqueConstraints = { @UniqueConstraint(columnNames = { "name", "version" }) }) -public class ParameterSet extends UniqueNameVersionPipelineComponent - implements Groupable { +public class ParameterSet extends UniqueNameVersionPipelineComponent { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_ParameterSet_generator") @SequenceGenerator(name = "ziggy_ParameterSet_generator", initialValue = 1, diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java index 86f6c90..4f14cc2 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java @@ -9,7 +9,7 @@ import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.hibernate.annotations.Cascade; import org.hibernate.annotations.CascadeType; import org.slf4j.Logger; @@ -53,8 +53,7 @@ @Entity @Table(name = "ziggy_PipelineDefinition", uniqueConstraints = { @UniqueConstraint(columnNames = { "name", "version" }) }) -public class PipelineDefinition extends UniqueNameVersionPipelineComponent - implements Groupable { +public class PipelineDefinition extends UniqueNameVersionPipelineComponent { @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(PipelineDefinition.class); @@ -205,6 +204,7 @@ public void setDescription(String description) { this.description = description; } + @Override public Long getId() { return id; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionExporter.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionExporter.java index 32e1230..245adf2 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionExporter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionExporter.java @@ -39,7 +39,7 @@ public void exportPipelineConfiguration(List pipelines, "destinationPath exists and is a directory: " + destinationFile); } - log.info("Exporting " + pipelines.size() + " pipelines to: " + destinationFile); + log.info("Exporting {} pipelines to: {}", pipelines.size(), destinationFile); PipelineDefinitionFile pipelineDefinitionFile = new PipelineDefinitionFile(); populatePiplineDefinitionFile(pipelines, pipelineDefinitionFile); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporter.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporter.java index 74be962..b52238b 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporter.java @@ -10,7 +10,7 @@ import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -77,27 +77,27 @@ public void importPipelineConfiguration(Collection files) { List pipelineDefinitions = new ArrayList<>(); List pipelineModuleDefinitions = new ArrayList<>(); for (File sourceFile : files) { - log.info("Unmarshalling pipeline configuration from: " + sourceFile + "..."); + log.info("Unmarshalling pipeline configuration from {}", sourceFile); PipelineDefinitionFile pipelineDefinitionFile = xmlManager.unmarshal(sourceFile); pipelineDefinitions.addAll(pipelineDefinitionFile.getPipelines()); pipelineModuleDefinitions.addAll(pipelineDefinitionFile.getModules()); - log.info("Unmarshalling pipeline configuration from: " + sourceFile + "...done"); + log.info("Unmarshalling pipeline configuration from {}...done", sourceFile); } List pipelineModuleNames = pipelineModuleDefinitions.stream() .map(PipelineModuleDefinition::getName) .collect(Collectors.toList()); - log.debug("Importing Module Definitions..."); + log.debug("Importing module definitions"); Map resourcesByModule = importModules( pipelineModuleDefinitions); - log.debug("Importing Module Definitions...done"); + log.debug("Importing module definitions...done"); - log.debug("Importing Pipeline Definitions"); + log.debug("Importing pipeline definitions"); Map> resourcesByPipelineDefinition = importPipelines( pipelineDefinitions, pipelineModuleNames); - log.debug("DONE importing pipeline configuration"); + log.debug("Importing pipeline definitions...done"); if (!dryRun) { - log.debug("Persisting module and pipeline definitions..."); + log.debug("Persisting module and pipeline definitions"); pipelineImportOperations().persistDefinitions(resourcesByModule, resourcesByPipelineDefinition); log.debug("Persisting module and pipeline definitions...done"); @@ -186,26 +186,26 @@ private Map> i Map nodesByName = new HashMap<>(); if (nodes.isEmpty()) { - log.error("Pipeline " + pipelineName + " has no nodes"); + log.error("Pipeline {} has no nodes", pipelineName); } for (PipelineDefinitionNode node : nodes) { String childNodeIds = node.getChildNodeNames() != null ? node.getChildNodeNames() : ""; - log.info("Adding node " + node.getModuleName() + " for pipeline " + pipelineName - + " with child node(s) " + childNodeIds); + log.info("Adding node {} for pipeline {} with child node(s) {}", + node.getModuleName(), pipelineName, childNodeIds); nodesByName.put(node.getModuleName(), node); } String rootNodeString = newPipelineDefinition.getRootNodeNames(); if (rootNodeString != null) { - log.info("Pipeline " + pipelineName + " root node names: " + rootNodeString); + log.info("Pipeline {} root node names are {}", pipelineName, rootNodeString); } else { - log.error("Pipeline " + pipelineName + " root node names string is null"); + log.error("Pipeline {} root node names string is null", pipelineName); } // pipeline-level parameters - log.info("Parameter sets for pipeline " + pipelineName + ": " - + newPipelineDefinition.parameterSetNamesFromXml().toString()); + log.info("Parameter sets for pipeline {} are {}", pipelineName, + newPipelineDefinition.parameterSetNamesFromXml().toString()); Set missingParameterSets = ZiggyCollectionUtils.elementsNotInSuperset( existingParameterSetNames, newPipelineDefinition.parameterSetNamesFromXml()); if (!CollectionUtils.isEmpty(missingParameterSets)) { @@ -250,8 +250,8 @@ private void addNodes(String pipelineName, List rootNodeNames, xmlNode.setPipelineName(pipelineName); // Module-level parameters - log.info("Parameter sets for node " + nodeName + ": " - + xmlNode.getXmlParameterSetNames().toString()); + log.info("Parameter sets for node {} are {}", nodeName, + xmlNode.getXmlParameterSetNames().toString()); Set missingParameterSets = ZiggyCollectionUtils.elementsNotInSuperset( existingParameterSetNames, xmlNode.getXmlParameterSetNames()); if (!CollectionUtils.isEmpty(missingParameterSets)) { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineExecutionTime.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineExecutionTime.java deleted file mode 100644 index cfa2fcc..0000000 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineExecutionTime.java +++ /dev/null @@ -1,114 +0,0 @@ -package gov.nasa.ziggy.pipeline.definition; - -import java.util.Date; - -import org.apache.commons.lang3.time.DurationFormatUtils; - -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; - -/** - * Provides capabilities needed by classes that track pipeline execution times, such as - * {@link PipelineTask} and {@link PipelineInstance}. Four pieces of information are managed: - *

    - *
  1. The start time of the very first processing attempt (a {@link Date}) - *
  2. The start time of the current processing attempt (in milliseconds since the start of Linux - * time) - *
  3. the total elapsed execution time for all prior processing attempts, in milliseconds - *
  4. The end time of the most recent processing attempt (a {@link Date}). - *
- *

- * In addition to getters and setters for the above values, classes that implement - * {@link PipelineExecutionTime} are provided with additional, higher-level methods that start and - * stop the execution "clock," and which calculate the current total elapsed processing time. - * - * @author PT - */ -public interface PipelineExecutionTime { - - void setStartProcessingTime(Date date); - - Date getStartProcessingTime(); - - void setEndProcessingTime(Date date); - - Date getEndProcessingTime(); - - void setPriorProcessingExecutionTimeMillis(long executionTimeMillis); - - long getPriorProcessingExecutionTimeMillis(); - - void setCurrentExecutionStartTimeMillis(long linuxStartTimeCurrentExecution); - - long getCurrentExecutionStartTimeMillis(); - - /** - * Starts execution timing for an instance. At start time, the processing-start time is set if - * currently null, and the start time for the current execution time is also set. - *

- * Users are advised not to call this method directly. The {@link PipelineTaskOperations} class will - * call this method when a change in state of a {@link PipelineExecutionTime} implementation - * requires it. - */ - default void startExecutionClock() { - - // Only start the clock if it's not currently running; this is indicated by the - // current execution start time being an invalid value. - if (getCurrentExecutionStartTimeMillis() <= 0) { - if (getStartProcessingTime() == null || getStartProcessingTime().getTime() <= 0) { - setStartProcessingTime(new Date()); - } - setCurrentExecutionStartTimeMillis(currentTimeMillis()); - } - } - - /** - * Stops execution timing for an instance. The current-execution start time is set to -1 and the - * elapsed time from the current execution attempt is added to the total execution time. The end - * processing time is also set. - *

- * Users are advised not to call this method directly. The {@link PipelineTaskOperations} class will - * call this method when a change in state of a {@link PipelineExecutionTime} implementation - * requires it. - */ - default void stopExecutionClock() { - - // Only stop the clock if it's currently running; this is indicated by the - // current execution start time being valid. - if (getCurrentExecutionStartTimeMillis() > 0) { - setEndProcessingTime(new Date()); - setPriorProcessingExecutionTimeMillis(getPriorProcessingExecutionTimeMillis() - + getEndProcessingTime().getTime() - getCurrentExecutionStartTimeMillis()); - setCurrentExecutionStartTimeMillis(-1); - } - } - - /** - * Calculates the total execution time for all processing attempts to date, in milliseconds. The - * total time includes the sum of the total execution time for all prior efforts; if a - * processing effort is current underway, the duration of the current effort is added to the - * total execution time from all prior efforts. - */ - default long totalExecutionTimeAllAttemptsMillis() { - long totalExecutionTime = getPriorProcessingExecutionTimeMillis(); - long startTimeCurrentExecutionMillis = getCurrentExecutionStartTimeMillis(); - if (startTimeCurrentExecutionMillis >= 0) { - totalExecutionTime += currentTimeMillis() - startTimeCurrentExecutionMillis; - } - return totalExecutionTime; - } - - // The current time in milliseconds since the start of the Unix era is obtained - // via a method for testing purposes. - default long currentTimeMillis() { - return System.currentTimeMillis(); - } - - default String elapsedTime() { - - if (getStartProcessingTime().getTime() == 0) { - return "-"; - } - return DurationFormatUtils.formatDuration(totalExecutionTimeAllAttemptsMillis(), - "HH:mm:ss"); - } -} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java index 37e48c4..fc7d51a 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java @@ -12,6 +12,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import jakarta.persistence.Embedded; import jakarta.persistence.Entity; import jakarta.persistence.EnumType; import jakarta.persistence.Enumerated; @@ -40,10 +41,12 @@ * stored in the database * * @author Todd Klaus + * @author PT + * @author Bill Wohler */ @Entity @Table(name = "ziggy_PipelineInstance") -public class PipelineInstance implements PipelineExecutionTime { +public class PipelineInstance { @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(PipelineInstance.class); @@ -114,19 +117,15 @@ public Priority unmarshal(String priority) throws Exception { */ private String name; - /** Timestamp that processing started on this pipeline instance */ - private Date startProcessingTime = new Date(0); - - /** Timestamp that processing ended (successfully) on this pipeline instance */ - private Date endProcessingTime = new Date(0); - - private long currentExecutionStartTimeMillis = -1; - - private long priorProcessingExecutionTimeMillis; - @ManyToOne private PipelineDefinition pipelineDefinition; + private Date created = new Date(); + + /** Elapsed execution time for the instance. */ + @Embedded + private ExecutionClock executionClock = new ExecutionClock(); + @Enumerated(EnumType.STRING) private State state = State.INITIALIZED; @@ -179,14 +178,6 @@ public PipelineInstance(PipelineDefinition pipeline) { pipelineDefinition = pipeline; } - public State getState() { - return state; - } - - public void setState(State state) { - this.state = state; - } - public Long getId() { return id; } @@ -195,6 +186,14 @@ public void setId(Long id) { this.id = id; } + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + public PipelineDefinition getPipelineDefinition() { return pipelineDefinition; } @@ -203,37 +202,36 @@ public void setPipelineDefinition(PipelineDefinition pipeline) { pipelineDefinition = pipeline; } - public Priority getPriority() { - return priority; + public Date getCreated() { + return created; } - public void setPriority(Priority priority) { - this.priority = priority; + public ExecutionClock getExecutionClock() { + return executionClock; } - @Override - public String toString() { - return ReflectionToStringBuilder.toString(this); + public void startExecutionClock() { + executionClock.start(); } - @Override - public Date getEndProcessingTime() { - return endProcessingTime; + public void stopExecutionClock() { + executionClock.stop(); } - @Override - public void setEndProcessingTime(Date endProcessingTime) { - this.endProcessingTime = endProcessingTime; + public State getState() { + return state; } - @Override - public Date getStartProcessingTime() { - return startProcessingTime; + public void setState(State state) { + this.state = state; } - @Override - public void setStartProcessingTime(Date startProcessingTime) { - this.startProcessingTime = startProcessingTime; + public Priority getPriority() { + return priority; + } + + public void setPriority(Priority priority) { + this.priority = priority; } public List getRootNodes() { @@ -260,23 +258,6 @@ public void addPipelineInstanceNode(PipelineInstanceNode pipelineInstanceNode) { pipelineInstanceNodes.add(pipelineInstanceNode); } - @Override - public int hashCode() { - return Objects.hash(id); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - PipelineInstance other = (PipelineInstance) obj; - return Objects.equals(id, other.id); - } - public Set getParameterSets() { return parameterSets; } @@ -289,14 +270,6 @@ public void addParameterSet(ParameterSet parameterSet) { parameterSets.add(parameterSet); } - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - public PipelineInstanceNode getStartNode() { return startNode; } @@ -328,23 +301,46 @@ public String getModelDescription(String modelType) { return modelMetadata.getModelDescription(); } - @Override - public void setPriorProcessingExecutionTimeMillis(long priorProcessingExecutionTimeMillis) { - this.priorProcessingExecutionTimeMillis = priorProcessingExecutionTimeMillis; + /** Generate a hash code for mutable fields in addition to the ID. */ + public int totalHashCode() { + return Objects.hash(executionClock, id, state); + } + + /** + * Returns true if the given object is equal to this object considering mutable fields in + * addition to the ID. + */ + public boolean totalEquals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineInstance other = (PipelineInstance) obj; + return Objects.equals(executionClock, other.executionClock) && Objects.equals(id, other.id) + && state == other.state; } @Override - public long getPriorProcessingExecutionTimeMillis() { - return priorProcessingExecutionTimeMillis; + public int hashCode() { + return Objects.hash(id); } @Override - public void setCurrentExecutionStartTimeMillis(long currentExecutionStartTimeMillis) { - this.currentExecutionStartTimeMillis = currentExecutionStartTimeMillis; + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineInstance other = (PipelineInstance) obj; + return Objects.equals(id, other.id); } @Override - public long getCurrentExecutionStartTimeMillis() { - return currentExecutionStartTimeMillis; + public String toString() { + return ReflectionToStringBuilder.toString(this); } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java index 59eee40..bba80b1 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java @@ -14,7 +14,8 @@ import gov.nasa.ziggy.metrics.Metric; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.util.ZiggyStringUtils; @@ -38,6 +39,7 @@ public abstract class PipelineModule { protected RunMode runMode; private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); /** * Standard constructor. Stores the {@link PipelineTask} and {@link RunMode} for the instance. @@ -47,10 +49,6 @@ public PipelineModule(PipelineTask pipelineTask, RunMode runMode) { this.runMode = checkNotNull(runMode, "runMode"); } - public final long taskId() { - return pipelineTask.getId(); - } - /** * Defines the subset of {@link ProcessingStep} enumerations that are valid, and the order in * which they are stepped through. @@ -86,7 +84,7 @@ public ProcessingStep nextProcessingStep(ProcessingStep currentProcessingStep) { * step in the database. */ public void incrementProcessingStep() { - pipelineTaskOperations().updateProcessingStep(taskId(), + pipelineTaskDataOperations().updateProcessingStep(pipelineTask, nextProcessingStep(currentProcessingStep())); } @@ -96,7 +94,7 @@ public void incrementProcessingStep() { * @return current processing step */ public ProcessingStep currentProcessingStep() { - return pipelineTaskOperations().pipelineTask(taskId()).getProcessingStep(); + return pipelineTaskDataOperations().processingStep(pipelineTask); } /** @@ -177,7 +175,7 @@ public final void processingCompleteTaskAction() { public abstract boolean processTask() throws Exception; /** - * Update the PipelineTask.summaryMetrics. + * Update the PipelineTask.pipelineTaskMetrics. *

* This default implementation adds a single category ("ALL") with the overall execution time. *

@@ -187,10 +185,10 @@ public final void processingCompleteTaskAction() { */ public void updateMetrics(PipelineTask pipelineTask, Map threadMetrics, long overallExecTimeMillis) { - List taskMetrics = new ArrayList<>(); - PipelineTaskMetrics m = new PipelineTaskMetrics("All", overallExecTimeMillis, Units.TIME); + List taskMetrics = new ArrayList<>(); + PipelineTaskMetric m = new PipelineTaskMetric("All", overallExecTimeMillis, Units.TIME); taskMetrics.add(m); - pipelineTask.setSummaryMetrics(taskMetrics); + pipelineTaskDataOperations().updatePipelineTaskMetrics(pipelineTask, taskMetrics); } public abstract String getModuleName(); @@ -228,18 +226,21 @@ protected List defaultRestartModes() { protected abstract List restartModes(); protected void logTaskInfo(PipelineInstance instance, PipelineTask task) { - log.debug("[" + getModuleName() + "]instance id = " + instance.getId()); - log.debug("[" + getModuleName() + "]instance node id = " + task.getId()); - log.debug( - "[" + getModuleName() + "]instance node uow = " + task.uowTaskInstance().briefState()); + log.debug("[{}]instance id = {}", getModuleName(), instance.getId()); + log.debug("[{}]instance node id = {}", getModuleName(), task); + log.debug("[{}]instance node uow = {}", getModuleName(), task.getUnitOfWork().briefState()); } - // TODO Make protected so it's not in the public API + // TODO Make the following protected so they're not in the public API // The unit test that uses this method can define its own PipelineModule to mock this field. public PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + public PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + // Run modes: // // The RunMode enum, below, must have all of the restart modes that can be used by any @@ -263,10 +264,6 @@ protected void resumeCurrentStep() { protected void resubmit() { } - /** Defines what it means to resume monitoring. The default action is to do nothing. */ - protected void resumeMonitoring() { - } - /** * Defines the standard run mode. The default action is to do nothing, so you'll definitely want * to override this method if you want your module to do something useful. @@ -278,7 +275,6 @@ public enum RunMode { RESTART_FROM_BEGINNING(PipelineModule::restartFromBeginning), RESUME_CURRENT_STEP(PipelineModule::resumeCurrentStep), RESUBMIT(PipelineModule::resubmit), - RESUME_MONITORING(PipelineModule::resumeMonitoring), STANDARD(PipelineModule::runStandard); private Consumer taskAction; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java index 0f6dff5..a5ca5f4 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java @@ -1,68 +1,50 @@ package gov.nasa.ziggy.pipeline.definition; import static com.google.common.base.Preconditions.checkState; -import static gov.nasa.ziggy.ui.util.HtmlBuilder.htmlBuilder; -import java.text.MessageFormat; -import java.util.ArrayList; +import java.io.Serializable; import java.util.Date; -import java.util.HashSet; -import java.util.List; import java.util.Objects; -import java.util.Set; import java.util.regex.Matcher; import java.util.regex.Pattern; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.uow.UnitOfWork; import gov.nasa.ziggy.uow.UnitOfWorkGenerator; -import gov.nasa.ziggy.util.HostNameUtils; -import jakarta.persistence.ElementCollection; +import jakarta.persistence.Embedded; import jakarta.persistence.Entity; -import jakarta.persistence.EnumType; -import jakarta.persistence.Enumerated; -import jakarta.persistence.FetchType; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; -import jakarta.persistence.JoinTable; -import jakarta.persistence.OrderColumn; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; /** - * Represents a single pipeline unit of work Associated with a{@link PipelineInstance}, a + * Represents a single pipeline unit of work associated with a {@link PipelineInstance}, a * {@link PipelineDefinitionNode} (which is associated with a {@link PipelineModuleDefinition}), and * a {@link UnitOfWorkGenerator} that represents the unit of work. *

+ * This class is immutable and is a suitable handle for code that operates on a pipeline task. + * Mutable fields associated with a {@code PipelineTask} are found in {@code PipelineTaskData}. + *

* Note that the {@link #equals(Object)} and {@link #hashCode()} methods are written in terms of * just the {@code id} field so this object should not be used in sets and maps until it has been * stored in the database * * @author Todd Klaus + * @author PT + * @author Bill Wohler */ @Entity @Table(name = "ziggy_PipelineTask") -public class PipelineTask implements PipelineExecutionTime { +public class PipelineTask implements Serializable, Comparable { @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(PipelineTask.class); - /** - * Log filename format. This is used by {@link MessageFormat} to produce the filename for a log - * file. The first element is the basename, "instanceId-taskId-moduleName" (i.e., - * "100-200-foo"). The second element is a job index, needed when running tasks on a remote - * system that can produce multiple log files per task (i.e., one per remote job). The final - * element is the task log index, a value that increments as a task gets executed or rerun, - * which allows the logs to be sorted into the order in which they were generated. - */ - private static final String LOG_FILENAME_FORMAT = "{0}.{1}-{2}.log"; - - private static final String ERROR_PREFIX = "ERROR - "; + private static final long serialVersionUID = 20240909L; @Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_PipelineTask_generator") @@ -70,86 +52,17 @@ public class PipelineTask implements PipelineExecutionTime { sequenceName = "ziggy_PipelineTask_sequence", allocationSize = 1) private Long id; - /** Current processing step of this task. */ - @Enumerated(EnumType.STRING) - private ProcessingStep processingStep = ProcessingStep.INITIALIZING; - - /** - * Task failed. The transition logic will not run and the pipeline will stall. For tasks with - * subtasks, this means that the number of failed subtasks exceeds the user-set threshold. - */ - private boolean error; - - /** Total number of subtasks. */ - private int totalSubtaskCount; - - /** Number of completed subtasks. */ - private int completedSubtaskCount; + private long pipelineInstanceId; - /** Number of failed subtasks. */ - private int failedSubtaskCount; + private String moduleName; - /** - * Flag that indicates that this task was re-run from the console after an error - */ - private boolean retry = false; + private String executableName; /** Timestamp this task was created (either by launcher or transition logic) */ private Date created = new Date(); - /** hostname of the worker that processed (or is processing) this task */ - private String workerHost; - - /** worker thread number that processed (or is processing) this task */ - private int workerThread; - - /** SVN revision of the build at the time this task was executed */ - private String softwareRevision; - - /** Timestamp that processing started on this task */ - private Date startProcessingTime = new Date(0); - - /** Timestamp that processing ended (success or failure) on this task */ - private Date endProcessingTime = new Date(0); - - /** Number of times this task failed and was rolled back */ - private int failureCount = 0; - - /** Count of the number of automatic resubmits performed for this task. */ - private int autoResubmitCount = 0; - - /** Indicates whether the task has been submitted for remote execution. */ - @Enumerated(EnumType.STRING) - private AlgorithmType processingMode; - - @ElementCollection(fetch = FetchType.EAGER) - @JoinTable(name = "ziggy_PipelineTask_uowTaskParameters") - private Set uowTaskParameters; - - @ElementCollection - @JoinTable(name = "ziggy_PipelineTask_summaryMetrics") - private List summaryMetrics = new ArrayList<>(); - - @ElementCollection - @OrderColumn(name = "idx") - @JoinTable(name = "ziggy_PipelineTask_execLog") - private List execLog = new ArrayList<>(); - - @ElementCollection - @JoinTable(name = "ziggy_PipelineTask_remoteJobs") - private Set remoteJobs = new HashSet<>(); - - private int taskLogIndex = 0; - - private long priorProcessingExecutionTimeMillis = -1; - - private long currentExecutionStartTimeMillis; - - private String moduleName; - - private String executableName; - - private long pipelineInstanceId; + @Embedded + private UnitOfWork unitOfWork; /** * Required by Hibernate @@ -158,7 +71,7 @@ public PipelineTask() { } public PipelineTask(PipelineInstance pipelineInstance, - PipelineInstanceNode pipelineInstanceNode) { + PipelineInstanceNode pipelineInstanceNode, UnitOfWork unitOfWork) { // The pipelineInstanceNode can be null in tests. if (pipelineInstanceNode != null) { @@ -170,235 +83,41 @@ public PipelineTask(PipelineInstance pipelineInstance, if (pipelineInstance != null && pipelineInstance.getId() != null) { pipelineInstanceId = pipelineInstance.getId(); } - } - - /** - * A human readable description of this task and its parameters. - */ - public String prettyPrint() { - - StringBuilder bldr = new StringBuilder(); - bldr.append("TaskId: ") - .append(getId()) - .append(" ") - .append("Module Software Revision: ") - .append(getSoftwareRevision()) - .append(" ") - .append(getModuleName()) - .append(" ") - .append(" UoW: ") - .append(uowTaskInstance().briefState()) - .append(" "); - - return bldr.toString(); - } - - @Override - public String toString() { - return "PipelineTask [id=" + id + ", processingStep=" + processingStep + ", uowTask=" - + uowTaskInstance().briefState() + "]"; - } - - public UnitOfWork uowTaskInstance() { - UnitOfWork uow = new UnitOfWork(); - uow.setParameters(uowTaskParameters); - return uow; - } - - public static String taskBaseName(long instanceId, long taskId, String moduleName) { - return baseNamePrefix(instanceId, taskId) + "-" + moduleName; - } - public static String baseNamePrefix(long instanceId, long taskId) { - return instanceId + "-" + taskId; + this.unitOfWork = unitOfWork; } public String taskBaseName() { - return taskBaseName(pipelineInstanceId, getId(), getModuleName()); - } - - /** - * Produces a file name for log files of the following form: {@code - * --.-.log - * } - * - * @param jobIndex index for the current job. Each pipeline task's algorithm execution can be - * performed across multiple independent jobs; this index identifies a specific job out of the - * set that are running for the current task. - */ - public String logFilename(int jobIndex) { - return MessageFormat.format(LOG_FILENAME_FORMAT, taskBaseName(), jobIndex, taskLogIndex); - } - - /** Returns the label for a task that is used in some UI displays. */ - public String taskLabelText() { - return htmlBuilder().appendBold("ID: ") - .append(getPipelineInstanceId()) - .append(":") - .append(getId()) - .appendBold(" WORKER: ") - .append(getWorkerName()) - .appendBold(" TASK: ") - .append(getModuleName()) - .append(" [") - .append(uowTaskInstance().briefState()) - .append("] ") - .appendBoldColor(getDisplayProcessingStep(), isError() ? "red" : "green") - .append(" ") - .appendItalic(elapsedTime()) - .toString(); + return getPipelineInstanceId() + "-" + getId() + "-" + getModuleName(); } - public ProcessingStep getProcessingStep() { - return processingStep; - } - - public String getDisplayProcessingStep() { - return (isError() ? ERROR_PREFIX : "") + getProcessingStep(); - } - - /** - * Use of this method is not recommended. Use - * {@link PipelineTaskOperations#updateProcessingStep(PipelineTask, ProcessingStep)} instead. - */ - public void setProcessingStep(ProcessingStep processingStep) { - this.processingStep = processingStep; - } - - public boolean isError() { - return error; - } - - public void setError(boolean error) { - this.error = error; - } - - public void clearError() { - setError(false); - } - - public int getTotalSubtaskCount() { - return totalSubtaskCount; - } - - public void setTotalSubtaskCount(int totalSubtaskCount) { - this.totalSubtaskCount = totalSubtaskCount; - } - - public int getCompletedSubtaskCount() { - return completedSubtaskCount; - } - - public void setCompletedSubtaskCount(int completedSubtaskCount) { - this.completedSubtaskCount = completedSubtaskCount; - } - - public int getFailedSubtaskCount() { - return failedSubtaskCount; - } - - public void setFailedSubtaskCount(int failedSubtaskCount) { - this.failedSubtaskCount = failedSubtaskCount; + public Long getId() { + return id; } - @Override - public Date getEndProcessingTime() { - return endProcessingTime; + public long getPipelineInstanceId() { + return pipelineInstanceId; } - @Override - public void setEndProcessingTime(Date endProcessingTime) { - this.endProcessingTime = endProcessingTime; + public String getModuleName() { + return moduleName; } - public int getFailureCount() { - return failureCount; + public String getExecutableName() { + return executableName; } - public void setFailureCount(int failureCount) { - this.failureCount = failureCount; + public Date getCreated() { + return created; } - public void incrementFailureCount() { - failureCount++; + public UnitOfWork getUnitOfWork() { + return unitOfWork; } @Override - public Date getStartProcessingTime() { - return startProcessingTime; - } - - @Override - public void setStartProcessingTime(Date startProcessingTime) { - this.startProcessingTime = startProcessingTime; - } - - public String getWorkerName() { - if (workerHost != null && workerHost.length() > 0) { - String host = HostNameUtils.callerHostNameOrLocalhost(workerHost); - return host + ":" + workerThread; - } - return "-"; - } - - public String getSoftwareRevision() { - return softwareRevision; - } - - public void setSoftwareRevision(String softwareRevision) { - this.softwareRevision = softwareRevision; - } - - public String getWorkerHost() { - return workerHost; - } - - public void setWorkerHost(String workerHost) { - this.workerHost = workerHost; - } - - public int getWorkerThread() { - return workerThread; - } - - public void setWorkerThread(int workerThread) { - this.workerThread = workerThread; - } - - public boolean isRetry() { - return retry; - } - - public void setRetry(boolean retry) { - this.retry = retry; - } - - public List getSummaryMetrics() { - return summaryMetrics; - } - - public void setSummaryMetrics(List summaryMetrics) { - this.summaryMetrics = summaryMetrics; - } - - public List getExecLog() { - return execLog; - } - - public void setExecLog(List execLog) { - this.execLog = execLog; - } - - public void setTaskLogIndex(int taskLogIndex) { - this.taskLogIndex = taskLogIndex; - } - - public int getTaskLogIndex() { - return taskLogIndex; - } - - public void incrementTaskLogIndex() { - taskLogIndex++; + public int compareTo(PipelineTask o) { + return (int) (getId() - o.getId()); } @Override @@ -418,6 +137,16 @@ public boolean equals(Object obj) { return Objects.equals(id, other.id); } + public String toFullString() { + return "IID=" + getPipelineInstanceId() + ", TID=" + getId() + ", M=" + getModuleName() + + ", UOW=" + getUnitOfWork().briefState(); + } + + @Override + public String toString() { + return getId().toString(); + } + public static final class TaskBaseNameMatcher { public static final int INSTANCE_ID_GROUP = 1; public static final int TASK_ID_GROUP = 2; @@ -426,13 +155,13 @@ public static final class TaskBaseNameMatcher { public static final String regex = "(\\d+)-(\\d+)-([^\\s\\.]+){1}?(\\.(\\d+))?"; public static final Pattern pattern = Pattern.compile(regex); + private String baseName; + private boolean matches; private long instanceId; private long taskId; private String moduleName; private boolean hasJobIndex; private int jobIndex; - private boolean matches; - private String baseName; public TaskBaseNameMatcher(String baseName) { this.baseName = baseName; @@ -453,14 +182,6 @@ public boolean matches() { return matches; } - private void checkMatch() { - checkState(matches, "String " + baseName + " does not match task base name pattern"); - } - - private void checkHasJobIndex() { - checkState(matches, "String " + baseName + " does not have a job index"); - } - public long instanceId() { checkMatch(); return instanceId; @@ -485,117 +206,13 @@ public int jobIndex() { checkHasJobIndex(); return jobIndex; } - } - - @Override - public void setPriorProcessingExecutionTimeMillis(long priorProcessingExecutionTimeMillis) { - this.priorProcessingExecutionTimeMillis = priorProcessingExecutionTimeMillis; - } - - @Override - public long getPriorProcessingExecutionTimeMillis() { - return priorProcessingExecutionTimeMillis; - } - - @Override - public void setCurrentExecutionStartTimeMillis(long currentExecutionStartTimeMillis) { - this.currentExecutionStartTimeMillis = currentExecutionStartTimeMillis; - } - @Override - public long getCurrentExecutionStartTimeMillis() { - return currentExecutionStartTimeMillis; - } - - public long getPipelineInstanceId() { - return pipelineInstanceId; - } - - public String getModuleName() { - return moduleName; - } - - public String getExecutableName() { - return executableName; - } - - public Set getRemoteJobs() { - return remoteJobs; - } - - public void setRemoteJobs(Set remoteJobs) { - this.remoteJobs = remoteJobs; - } - - public boolean isRemoteExecution() { - return processingMode.equals(AlgorithmType.REMOTE); - } - - public void setRemoteExecution(boolean remoteExecution) { - processingMode = remoteExecution ? AlgorithmType.REMOTE : AlgorithmType.LOCAL; - } - - public AlgorithmType getProcessingMode() { - return processingMode; - } - - public void setProcessingMode(AlgorithmType mode) { - processingMode = mode; - } - - public int getAutoResubmitCount() { - return autoResubmitCount; - } - - public void setAutoResubmitCount(int autoResubmitCount) { - this.autoResubmitCount = autoResubmitCount; - } - - public void incrementAutoResubmitCount() { - autoResubmitCount++; - } - - public void resetAutoResubmitCount() { - autoResubmitCount = 0; - } - - public Date getCreated() { - return created; - } - - public void setCreated(Date created) { - this.created = created; - } - - public Long getId() { - return id; - } - - public Set getUowTaskParameters() { - return uowTaskParameters; - } - - public void setUowTaskParameters(Set uowTaskParameters) { - this.uowTaskParameters = uowTaskParameters; - } - - /** - * Starts execution clock for {@link PipelineTask} by first using the default method from - * {@link PipelineExecutionTime}, then starting the clock for the {@link PipelineInstance} as - * well. Users are generally advised not to call this method directly, as the clock is started - * and stopped at appropriate times when a {@link PipelineTaskOperations} instance is used to - * set a task's state. - */ - @Override - public void startExecutionClock() { - PipelineExecutionTime.super.startExecutionClock(); - } + private void checkMatch() { + checkState(matches, "String " + baseName + " does not match task base name pattern"); + } - public double costEstimate() { - double estimate = 0; - for (RemoteJob job : remoteJobs) { - estimate += job.getCostEstimate(); + private void checkHasJobIndex() { + checkState(matches, "String " + baseName + " does not have a job index"); } - return estimate; } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskData.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskData.java new file mode 100644 index 0000000..ba2cdce --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskData.java @@ -0,0 +1,344 @@ +package gov.nasa.ziggy.pipeline.definition; + +import static com.google.common.base.Preconditions.checkNotNull; +import static com.google.common.base.Preconditions.checkState; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Objects; +import java.util.Set; + +import org.apache.commons.lang3.StringUtils; + +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataCrud; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.util.HostNameUtils; +import jakarta.persistence.ElementCollection; +import jakarta.persistence.Embedded; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.Id; +import jakarta.persistence.JoinTable; +import jakarta.persistence.OneToOne; +import jakarta.persistence.OrderColumn; +import jakarta.persistence.Table; + +/** + * Data for a given {@link PipelineTask}. + *

+ * The {@link PipelineTask} class contains only immutable fields. All mutable values or immutable + * collections are for a task are stored here. This is necessary because pipeline task data can + * change quickly, and we do not want a pipeline task in memory that contains obsolete information. + * Hence the pipeline task itself contains only immutable (hence valid) information, while all + * access to mutable data in this class occurs via the {@link PipelineTaskDataOperations} class. + *

+ * Note that the {@link #equals(Object)} and {@link #hashCode()} methods are written in terms of + * just the pipeline task's ID so this object should not be created until its associated + * {@link PipelineTask} has been stored in the database. The {@code setPipelineTask()} method is + * omitted by design. + *

+ * This class is not for public consumption. It should live ephemerally in + * {@link PipelineTaskDataCrud} and {@link PipelineTaskDataOperations} methods. + * + * @author PT + * @author Bill Wohler + */ +@Entity +@Table(name = "ziggy_PipelineTaskData") +public class PipelineTaskData implements Comparable { + + /** + * The database ID for {@link PipelineTaskData} is the ID of the corresponding + * {@link PipelineTask} instance. + */ + @Id + private long pipelineTaskId; + + /** The pipeline task associated with this object. */ + @OneToOne + private PipelineTask pipelineTask; + + /** Ziggy software revision at the time this task was executed. */ + private String ziggySoftwareRevision; + + /** Revision of the pipeline build at the time this task was executed. */ + private String pipelineSoftwareRevision; + + /** Host name of the worker that processed (or is processing) this task. */ + private String workerHost; + + /** Worker thread number that processed (or is processing) this task. */ + private int workerThread; + + /** Elapsed execution time for the task. */ + @Embedded + private ExecutionClock executionClock = new ExecutionClock(); + + /** Current processing step of this task. */ + @Enumerated(EnumType.STRING) + private ProcessingStep processingStep = ProcessingStep.WAITING_TO_RUN; + + /** + * Task failed. The transition logic will not run and the pipeline will stall. For tasks with + * subtasks, this means that the number of failed subtasks exceeds the user-set threshold. + */ + private boolean error; + + /** User has requested that task execution halt. */ + private boolean haltRequested; + + /** Total number of subtasks. */ + private int totalSubtaskCount; + + /** Number of completed subtasks. */ + private int completedSubtaskCount; + + /** Number of failed subtasks. */ + private int failedSubtaskCount; + + /** + * Flag that indicates that this task was re-run from the console after an error + */ + private boolean retry; + + /** Number of times this task failed and was rolled back. */ + private int failureCount; + + /** Count of the number of automatic resubmits performed for this task. */ + private int autoResubmitCount; + + /** Indicates whether the task has been submitted for remote execution. */ + @Enumerated(EnumType.STRING) + private AlgorithmType algorithmType; + + private int taskLogIndex; + + @ElementCollection + @OrderColumn(name = "idx") + @JoinTable(name = "ziggy_PipelineTaskData_pipelineTaskMetrics") + private List pipelineTaskMetrics = new ArrayList<>(); + + @ElementCollection + @OrderColumn(name = "idx") + @JoinTable(name = "ziggy_PipelineTaskData_taskExecutionLogs") + private List taskExecutionLogs = new ArrayList<>(); + + @ElementCollection + @JoinTable(name = "ziggy_PipelineTaskData_remoteJobs") + private Set remoteJobs = new HashSet<>(); + + /** For Hibernate use only. */ + PipelineTaskData() { + } + + /** + * Creates a {@code PipelineTaskData} object. The given non-null {@link PipelineTask} object + * must already have been persisted to the database. + */ + public PipelineTaskData(PipelineTask pipelineTask) { + checkNotNull(pipelineTask); + checkState(pipelineTask.getId() > 0); + this.pipelineTask = pipelineTask; + pipelineTaskId = pipelineTask.getId(); + } + + public PipelineTask getPipelineTask() { + return pipelineTask; + } + + public String getZiggySoftwareRevision() { + return ziggySoftwareRevision; + } + + public void setZiggySoftwareRevision(String ziggySoftwareRevision) { + this.ziggySoftwareRevision = ziggySoftwareRevision; + } + + public String getPipelineSoftwareRevision() { + return pipelineSoftwareRevision; + } + + public void setPipelineSoftwareRevision(String pipelineSoftwareRevision) { + this.pipelineSoftwareRevision = pipelineSoftwareRevision; + } + + public String getWorkerHost() { + return workerHost; + } + + public void setWorkerHost(String workerHost) { + this.workerHost = workerHost; + } + + public int getWorkerThread() { + return workerThread; + } + + public void setWorkerThread(int workerThread) { + this.workerThread = workerThread; + } + + public String workerName() { + if (StringUtils.isBlank(workerHost)) { + return "-"; + } + return HostNameUtils.callerHostNameOrLocalhost(workerHost) + ":" + workerThread; + } + + public ExecutionClock getExecutionClock() { + return executionClock; + } + + public ProcessingStep getProcessingStep() { + return processingStep; + } + + public void setProcessingStep(ProcessingStep processingStep) { + this.processingStep = processingStep; + } + + public boolean isError() { + return error; + } + + public void setError(boolean error) { + this.error = error; + } + + public boolean isHaltRequested() { + return haltRequested; + } + + public void setHaltRequested(boolean haltRequested) { + this.haltRequested = haltRequested; + } + + public int getTotalSubtaskCount() { + return totalSubtaskCount; + } + + public void setTotalSubtaskCount(int totalSubtaskCount) { + this.totalSubtaskCount = totalSubtaskCount; + } + + public int getCompletedSubtaskCount() { + return completedSubtaskCount; + } + + public void setCompletedSubtaskCount(int completedSubtaskCount) { + this.completedSubtaskCount = completedSubtaskCount; + } + + public int getFailedSubtaskCount() { + return failedSubtaskCount; + } + + public void setFailedSubtaskCount(int failedSubtaskCount) { + this.failedSubtaskCount = failedSubtaskCount; + } + + public boolean isRetry() { + return retry; + } + + public void setRetry(boolean retry) { + this.retry = retry; + } + + public int getFailureCount() { + return failureCount; + } + + public void setFailureCount(int failureCount) { + this.failureCount = failureCount; + } + + public void incrementFailureCount() { + failureCount++; + } + + public int getAutoResubmitCount() { + return autoResubmitCount; + } + + public void setAutoResubmitCount(int autoResubmitCount) { + this.autoResubmitCount = autoResubmitCount; + } + + public void incrementAutoResubmitCount() { + autoResubmitCount++; + } + + public void resetAutoResubmitCount() { + autoResubmitCount = 0; + } + + public AlgorithmType getAlgorithmType() { + return algorithmType; + } + + public void setAlgorithmType(AlgorithmType mode) { + algorithmType = mode; + } + + public void setTaskLogIndex(int taskLogIndex) { + this.taskLogIndex = taskLogIndex; + } + + public int getTaskLogIndex() { + return taskLogIndex; + } + + public void incrementTaskLogIndex() { + taskLogIndex++; + } + + public List getPipelineTaskMetrics() { + return pipelineTaskMetrics; + } + + public void setPipelineTaskMetrics(List pipelineTaskMetrics) { + this.pipelineTaskMetrics = pipelineTaskMetrics; + } + + public List getTaskExecutionLogs() { + return taskExecutionLogs; + } + + public void setTaskExecutionLogs(List taskExecutionLogs) { + this.taskExecutionLogs = taskExecutionLogs; + } + + public Set getRemoteJobs() { + return remoteJobs; + } + + public void setRemoteJobs(Set remoteJobs) { + this.remoteJobs = remoteJobs; + } + + @Override + public int compareTo(PipelineTaskData o) { + return (int) (pipelineTaskId - o.pipelineTaskId); + } + + @Override + public int hashCode() { + return Objects.hash(pipelineTaskId); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineTaskData other = (PipelineTaskData) obj; + return pipelineTaskId == other.pipelineTaskId; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDisplayData.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDisplayData.java new file mode 100644 index 0000000..9107dbd --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDisplayData.java @@ -0,0 +1,156 @@ +package gov.nasa.ziggy.pipeline.definition; + +import java.util.Date; +import java.util.List; +import java.util.Objects; +import java.util.Set; + +/** + * Projection of a mixture of fields from {@link PipelineTask} and its corresponding + * {@link PipelineTaskData} instance. + * + * @author PT + * @author Bill Wohler + */ +public class PipelineTaskDisplayData { + + private static final String ERROR_PREFIX = "ERROR - "; + + private PipelineTaskData pipelineTaskData; + + public PipelineTaskDisplayData(PipelineTaskData pipelineTaskData) { + this.pipelineTaskData = pipelineTaskData; + } + + public double costEstimate() { + double estimate = 0; + for (RemoteJob job : getRemoteJobs()) { + estimate += job.getCostEstimate(); + } + return estimate; + } + + public PipelineTask getPipelineTask() { + return pipelineTaskData.getPipelineTask(); + } + + public long getPipelineTaskId() { + return getPipelineTask().getId(); + } + + public long getPipelineInstanceId() { + return getPipelineTask().getPipelineInstanceId(); + } + + public Date getCreated() { + return getPipelineTask().getCreated(); + } + + public String getModuleName() { + return getPipelineTask().getModuleName(); + } + + public String getBriefState() { + return getPipelineTask().getUnitOfWork().briefState(); + } + + public String getZiggySoftwareRevision() { + return pipelineTaskData.getZiggySoftwareRevision(); + } + + public String getPipelineSoftwareRevision() { + return pipelineTaskData.getPipelineSoftwareRevision(); + } + + public String getWorkerName() { + return pipelineTaskData.workerName(); + } + + public ExecutionClock getExecutionClock() { + return pipelineTaskData.getExecutionClock(); + } + + public ProcessingStep getProcessingStep() { + return pipelineTaskData.getProcessingStep(); + } + + public String getDisplayProcessingStep() { + return (isError() ? ERROR_PREFIX : "") + getProcessingStep(); + } + + public boolean isError() { + return pipelineTaskData.isError(); + } + + public int getTotalSubtaskCount() { + return pipelineTaskData.getTotalSubtaskCount(); + } + + public int getCompletedSubtaskCount() { + return pipelineTaskData.getCompletedSubtaskCount(); + } + + public int getFailedSubtaskCount() { + return pipelineTaskData.getFailedSubtaskCount(); + } + + public int getFailureCount() { + return pipelineTaskData.getFailureCount(); + } + + public List getPipelineTaskMetrics() { + return pipelineTaskData.getPipelineTaskMetrics(); + } + + public Set getRemoteJobs() { + return pipelineTaskData.getRemoteJobs(); + } + + @Override + public int hashCode() { + return Objects.hash(getPipelineTaskId(), getPipelineInstanceId(), getCreated(), + getModuleName(), getBriefState(), getZiggySoftwareRevision(), + getPipelineSoftwareRevision(), getWorkerName(), getExecutionClock(), + getProcessingStep(), isError(), getTotalSubtaskCount(), getCompletedSubtaskCount(), + getFailedSubtaskCount(), getFailureCount(), getPipelineTaskMetrics(), getRemoteJobs()); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineTaskDisplayData other = (PipelineTaskDisplayData) obj; + return Objects.equals(pipelineTaskData, other.pipelineTaskData) + && Objects.equals(getPipelineTaskId(), other.getPipelineTaskId()) + && Objects.equals(getPipelineInstanceId(), other.getPipelineInstanceId()) + && Objects.equals(getCreated(), other.getCreated()) + && Objects.equals(getModuleName(), other.getModuleName()) + && Objects.equals(getBriefState(), other.getBriefState()) + && Objects.equals(getZiggySoftwareRevision(), other.getZiggySoftwareRevision()) + && Objects.equals(getPipelineSoftwareRevision(), other.getPipelineSoftwareRevision()) + && Objects.equals(getWorkerName(), other.getWorkerName()) + && Objects.equals(getExecutionClock(), other.getExecutionClock()) + && Objects.equals(getProcessingStep(), other.getProcessingStep()) + && Objects.equals(isError(), other.isError()) + && Objects.equals(getTotalSubtaskCount(), other.getTotalSubtaskCount()) + && Objects.equals(getCompletedSubtaskCount(), other.getCompletedSubtaskCount()) + && Objects.equals(getFailedSubtaskCount(), other.getFailedSubtaskCount()) + && Objects.equals(getFailureCount(), other.getFailureCount()) + && Objects.equals(getPipelineTaskMetrics(), other.getPipelineTaskMetrics()) + && Objects.equals(getRemoteJobs(), other.getRemoteJobs()); + } + + public String toFullString() { + return getClass().getSimpleName() + ": pipelineTaskId=" + getPipelineTaskId() + + ", moduleName=" + getModuleName() + ", briefState=" + getBriefState(); + } + + @Override + public String toString() { + return getPipelineTask().toString(); + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetrics.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetric.java similarity index 61% rename from src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetrics.java rename to src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetric.java index 05edf2c..2817fe9 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetrics.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskMetric.java @@ -1,5 +1,7 @@ package gov.nasa.ziggy.pipeline.definition; +import java.util.Objects; + import jakarta.persistence.Embeddable; import jakarta.persistence.EnumType; import jakarta.persistence.Enumerated; @@ -8,7 +10,7 @@ * @author Todd Klaus */ @Embeddable -public class PipelineTaskMetrics { +public class PipelineTaskMetric { public enum Units { TIME, BYTES, RATE } @@ -20,10 +22,10 @@ public enum Units { @Enumerated(EnumType.STRING) private Units units; - public PipelineTaskMetrics() { + public PipelineTaskMetric() { } - public PipelineTaskMetrics(String category, long value, Units units) { + public PipelineTaskMetric(String category, long value, Units units) { this.category = category; this.value = value; this.units = units; @@ -57,4 +59,22 @@ public void setUnits(Units units) { public String toString() { return "category: " + category + ", timeMillis: " + value + ", units: " + units; } + + @Override + public int hashCode() { + return Objects.hash(category, units, value); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineTaskMetric other = (PipelineTaskMetric) obj; + return Objects.equals(category, other.category) && units == other.units + && value == other.value; + } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStep.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStep.java index 9c8a01b..973e9a0 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStep.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStep.java @@ -1,5 +1,6 @@ package gov.nasa.ziggy.pipeline.definition; +import java.io.Serializable; import java.util.Set; import java.util.function.Consumer; @@ -17,7 +18,7 @@ * @author PT * @author Bill Wohler */ -public enum ProcessingStep { +public enum ProcessingStep implements Serializable { INITIALIZING(PipelineModule::initializingTaskAction, Counts::incrementInitializingTaskCount), WAITING_TO_RUN(PipelineModule::waitingToRunTaskAction, Counts::incrementWaitingToRunTaskCount), @@ -50,6 +51,11 @@ public boolean isInfrastructureStep() { return this == INITIALIZING || this == WAITING_TO_RUN || this == COMPLETE; } + public boolean isPreExecutionStep() { + return this == INITIALIZING || this == WAITING_TO_RUN || this == MARSHALING + || this == QUEUED || this == SUBMITTING; + } + public static Set processingSteps() { return Set.of(WAITING_TO_RUN, MARSHALING, SUBMITTING, QUEUED, EXECUTING, WAITING_TO_STORE, STORING); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/RemoteJob.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/RemoteJob.java index f80d1b6..d15c723 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/RemoteJob.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/RemoteJob.java @@ -28,7 +28,7 @@ */ @Embeddable -public class RemoteJob { +public class RemoteJob implements Comparable { private long jobId; private double costEstimate; @@ -82,6 +82,11 @@ public boolean equals(Object obj) { return jobId == other.jobId; } + @Override + public int compareTo(RemoteJob otherRemoteJob) { + return (int) (jobId - otherRemoteJob.jobId); + } + /** * Convenience class that transports a job's "select" value (model and node count) and wall time * value from qstat. diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskCounts.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskCounts.java index e32080b..1574033 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskCounts.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskCounts.java @@ -5,6 +5,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Objects; /** * Summary statistics for a collection of pipeline tasks. @@ -14,15 +15,123 @@ */ public class TaskCounts { - public class Counts { + private final int taskCount; + private final Map moduleCounts = new LinkedHashMap<>(); + private final Counts totalCounts = new Counts(); + private List moduleNames; + + public TaskCounts() { + taskCount = 0; + moduleNames = new ArrayList<>(); + } + + public TaskCounts(List tasks) { + taskCount = tasks.size(); + for (PipelineTaskDisplayData task : tasks) { + Counts counts = moduleCounts.get(task.getModuleName()); + if (counts == null) { + counts = new Counts(); + moduleCounts.put(task.getModuleName(), counts); + } + + if (!task.isError()) { + task.getProcessingStep().incrementTaskCount(counts); + } else { + counts.failedTaskCount++; + } + + counts.subtaskCounts.totalSubtaskCount += task.getTotalSubtaskCount(); + counts.subtaskCounts.completedSubtaskCount += task.getCompletedSubtaskCount(); + counts.subtaskCounts.failedSubtaskCount += task.getFailedSubtaskCount(); + } + + for (Counts counts : moduleCounts.values()) { + totalCounts.waitingToRunTaskCount += counts.getWaitingToRunTaskCount(); + totalCounts.processingTaskCount += counts.getProcessingTaskCount(); + totalCounts.completedTaskCount += counts.getCompletedTaskCount(); + totalCounts.failedTaskCount += counts.getFailedTaskCount(); + totalCounts.subtaskCounts.totalSubtaskCount += counts.getTotalSubtaskCount(); + totalCounts.subtaskCounts.completedSubtaskCount += counts.getCompletedSubtaskCount(); + totalCounts.subtaskCounts.failedSubtaskCount += counts.getFailedSubtaskCount(); + } + + moduleNames = new ArrayList<>(moduleCounts.keySet()); + } + + public static String subtaskCountsLabel(int completeCount, int totalCount, int failedCount) { + StringBuilder s = new StringBuilder().append(completeCount).append("/").append(totalCount); + if (failedCount > 0) { + s.append(" (").append(failedCount).append(")"); + } + return s.toString(); + } + + /** + * Returns true if getTaskCount() == getTotalCounts().getCompletedTaskCount(). + */ + public boolean isPipelineTasksComplete() { + return getTotalCounts().getCompletedTaskCount() == taskCount; + } + + /** + * Returns true if getTaskCount() == getTotalCounts().getCompletedTaskCount() + + * getTotalCounts().getFailedTaskCount() for positive task counts. + */ + public boolean isPipelineTasksExecutionComplete() { + return getTotalCounts().getCompletedTaskCount() + + getTotalCounts().getFailedTaskCount() == taskCount && taskCount > 0; + } + + public int getTaskCount() { + return taskCount; + } + + public List getModuleNames() { + return moduleNames; + } + + public Map getModuleCounts() { + return moduleCounts; + } + + public Counts getTotalCounts() { + return totalCounts; + } + + @Override + public int hashCode() { + return Objects.hash(moduleCounts, moduleNames, taskCount, totalCounts); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + TaskCounts other = (TaskCounts) obj; + return Objects.equals(moduleCounts, other.moduleCounts) + && Objects.equals(moduleNames, other.moduleNames) && taskCount == other.taskCount + && Objects.equals(totalCounts, other.totalCounts); + } + + @Override + public String toString() { + return MessageFormat.format( + "taskCount={0}, waitingToRunTaskCount={1}, completedTaskCount={2}, failedTaskCount={3}", + getTaskCount(), getTotalCounts().getWaitingToRunTaskCount(), + getTotalCounts().getCompletedTaskCount(), getTotalCounts().getFailedTaskCount()); + } + + public static class Counts { private int waitingToRunTaskCount; private int processingTaskCount; private int completedTaskCount; private int failedTaskCount; - private int completedSubtaskCount; - private int failedSubtaskCount; - private int totalSubtaskCount; + private SubtaskCounts subtaskCounts = new SubtaskCounts(); public String subtaskCountsLabel() { return TaskCounts.subtaskCountsLabel(getCompletedSubtaskCount(), getTotalSubtaskCount(), @@ -45,16 +154,16 @@ public int getFailedTaskCount() { return failedTaskCount; } - public int getCompletedSubtaskCount() { - return completedSubtaskCount; + public int getTotalSubtaskCount() { + return subtaskCounts.getTotalSubtaskCount(); } - public int getFailedSubtaskCount() { - return failedSubtaskCount; + public int getCompletedSubtaskCount() { + return subtaskCounts.getCompletedSubtaskCount(); } - public int getTotalSubtaskCount() { - return totalSubtaskCount; + public int getFailedSubtaskCount() { + return subtaskCounts.getFailedSubtaskCount(); } public void incrementInitializingTaskCount() { @@ -92,96 +201,74 @@ public void incrementStoringTaskCount() { public void incrementCompleteTaskCount() { completedTaskCount++; } - } - - private final int taskCount; - private final Map moduleCounts = new LinkedHashMap<>(); - private final Counts totalCounts = new Counts(); - private List moduleNames; - public TaskCounts() { - taskCount = 0; - moduleNames = new ArrayList<>(); - } + @Override + public int hashCode() { + return Objects.hash(completedTaskCount, failedTaskCount, processingTaskCount, + subtaskCounts, waitingToRunTaskCount); + } - public TaskCounts(List tasks) { - taskCount = tasks.size(); - for (PipelineTask task : tasks) { - Counts counts = moduleCounts.get(task.getModuleName()); - if (counts == null) { - counts = new Counts(); - moduleCounts.put(task.getModuleName(), counts); + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; } - - if (!task.isError()) { - task.getProcessingStep().incrementTaskCount(counts); - } else { - counts.failedTaskCount++; + if ((obj == null) || (getClass() != obj.getClass())) { + return false; } - - counts.totalSubtaskCount += task.getTotalSubtaskCount(); - counts.completedSubtaskCount += task.getCompletedSubtaskCount(); - counts.failedSubtaskCount += task.getFailedSubtaskCount(); - } - - for (Counts counts : moduleCounts.values()) { - totalCounts.waitingToRunTaskCount += counts.getWaitingToRunTaskCount(); - totalCounts.processingTaskCount += counts.getProcessingTaskCount(); - totalCounts.completedTaskCount += counts.getCompletedTaskCount(); - totalCounts.failedTaskCount += counts.getFailedTaskCount(); - totalCounts.totalSubtaskCount += counts.getTotalSubtaskCount(); - totalCounts.completedSubtaskCount += counts.getCompletedSubtaskCount(); - totalCounts.failedSubtaskCount += counts.getFailedSubtaskCount(); + Counts other = (Counts) obj; + return completedTaskCount == other.completedTaskCount + && failedTaskCount == other.failedTaskCount + && processingTaskCount == other.processingTaskCount + && Objects.equals(subtaskCounts, other.subtaskCounts) + && waitingToRunTaskCount == other.waitingToRunTaskCount; } - - moduleNames = new ArrayList<>(moduleCounts.keySet()); } - public static String subtaskCountsLabel(int completeCount, int totalCount, int failedCount) { - StringBuilder s = new StringBuilder().append(completeCount).append("/").append(totalCount); - if (failedCount > 0) { - s.append(" (").append(failedCount).append(")"); - } - return s.toString(); - } + public static class SubtaskCounts { + private int totalSubtaskCount; + private int completedSubtaskCount; + private int failedSubtaskCount; - /** - * Returns true if getTaskCount() == getTotalCounts().getCompletedTaskCount(). - */ - public boolean isPipelineTasksComplete() { - return getTotalCounts().getCompletedTaskCount() == taskCount; - } + public SubtaskCounts() { + } - /** - * Returns true if getTaskCount() == getTotalCounts().getCompletedTaskCount() + - * getTotalCounts().getFailedTaskCount() for positive task counts. - */ - public boolean isPipelineTasksExecutionComplete() { - return getTotalCounts().getCompletedTaskCount() - + getTotalCounts().getFailedTaskCount() == taskCount && taskCount > 0; - } + public SubtaskCounts(int totalSubtaskCount, int completedSubtaskCount, + int failedSubtaskCount) { + this.totalSubtaskCount = totalSubtaskCount; + this.completedSubtaskCount = completedSubtaskCount; + this.failedSubtaskCount = failedSubtaskCount; + } - public int getTaskCount() { - return taskCount; - } + public int getTotalSubtaskCount() { + return totalSubtaskCount; + } - public List getModuleNames() { - return moduleNames; - } + public int getCompletedSubtaskCount() { + return completedSubtaskCount; + } - public Map getModuleCounts() { - return moduleCounts; - } + public int getFailedSubtaskCount() { + return failedSubtaskCount; + } - public Counts getTotalCounts() { - return totalCounts; - } + @Override + public int hashCode() { + return Objects.hash(completedSubtaskCount, failedSubtaskCount, totalSubtaskCount); + } - @Override - public String toString() { - return MessageFormat.format( - "taskCount={0}, waitingToRunTaskCount={1}, completedTaskCount={2}, failedTaskCount={3}", - getTaskCount(), getTotalCounts().getWaitingToRunTaskCount(), - getTotalCounts().getCompletedTaskCount(), getTotalCounts().getFailedTaskCount()); + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + SubtaskCounts other = (SubtaskCounts) obj; + return completedSubtaskCount == other.completedSubtaskCount + && failedSubtaskCount == other.failedSubtaskCount + && totalSubtaskCount == other.totalSubtaskCount; + } } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/DataFileTypeCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/DataFileTypeCrud.java index df63ace..4c85133 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/DataFileTypeCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/DataFileTypeCrud.java @@ -57,12 +57,12 @@ public void persist(Collection collection) { .collect(Collectors.toList()); if (validInstances.size() < collection.size()) { int invalidInstanceCount = collection.size() - validInstances.size(); - log.info("Removed " + invalidInstanceCount + " from collection of objects to create"); + log.info("Removed {} from collection of objects to create", invalidInstanceCount); } createdInstances.addAll(validInstances); - log.info("Creating " + createdInstances.size() + " instances of DataFileType in database"); + log.info("Creating {} instances of DataFileType in database", createdInstances.size()); super.persist(createdInstances); - log.info("Created " + createdInstances.size() + " instances of DataFileType in database"); + log.info("Created {} instances of DataFileType in database", createdInstances.size()); } public DataFileType retrieveByName(String name) { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupCrud.java index 63eccdb..963612c 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupCrud.java @@ -9,15 +9,13 @@ import gov.nasa.ziggy.crud.AbstractCrud; import gov.nasa.ziggy.pipeline.definition.Group; import gov.nasa.ziggy.pipeline.definition.Group_; -import gov.nasa.ziggy.pipeline.definition.Groupable; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.services.database.DatabaseService; /** * Provides CRUD methods for {@link Group} * * @author Todd Klaus + * @author Bill Wohler */ public class GroupCrud extends AbstractCrud { @SuppressWarnings("unused") @@ -30,36 +28,26 @@ public GroupCrud(DatabaseService databaseService) { super(databaseService); } - public List retrieveAll() { - List groups = list( - createZiggyQuery(Group.class).column(Group_.name).ascendingOrder()); - return groups; - } - - public List retrieveAll(Class clazz) { - List groups = retrieveAll(); - for (Group group : groups) { - setGroupMemberNames(group, clazz); + public Group retrieve(String name, String type) { + if (StringUtils.isBlank(name) || StringUtils.isBlank(type)) { + return Group.DEFAULT; } - return groups; + Group group = uniqueResult(createZiggyQuery(Group.class).column(Group_.name) + .in(name) + .column(Group_.type) + .in(type)); + return group != null ? group : Group.DEFAULT; } - public Group retrieveGroupByName(String name, Class clazz) { - if (StringUtils.isBlank(name)) { - return Group.DEFAULT; - } - Group group = uniqueResult(createZiggyQuery(Group.class).column(Group_.name).in(name)); - setGroupMemberNames(group, clazz); - return group; + public List retrieveAll() { + return list(createZiggyQuery(Group.class).column(Group_.name).ascendingOrder()); } - private void setGroupMemberNames(Group group, Class clazz) { - if (clazz.equals(PipelineDefinition.class)) { - group.setMemberNames(group.getPipelineDefinitionNames()); - } - if (clazz.equals(ParameterSet.class)) { - group.setMemberNames(group.getParameterSetNames()); - } + public List retrieveAll(String type) { + return list(createZiggyQuery(Group.class).column(Group_.type) + .in(type) + .column(Group_.name) + .ascendingOrder()); } @Override diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperations.java index 81512e1..479af66 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperations.java @@ -5,7 +5,6 @@ import java.util.Set; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.services.database.DatabaseOperations; /** @@ -36,16 +35,16 @@ public Set merge(Set groups) { }); } - public List groups() { - return performTransaction(() -> groupCrud.retrieveAll()); + public Group group(String name, String type) { + return performTransaction(() -> groupCrud.retrieve(name, type)); } - public List groupsForClass(Class clazz) { - return performTransaction(() -> groupCrud.retrieveAll(clazz)); + public List groups() { + return performTransaction(() -> groupCrud.retrieveAll()); } - public Group groupForName(String name, Class clazz) { - return performTransaction(() -> groupCrud.retrieveGroupByName(name, clazz)); + public List groups(String type) { + return performTransaction(() -> groupCrud.retrieveAll(type)); } public void delete(Group group) { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/HasExternalIdCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/HasExternalIdCrud.java index c94d5cf..887f970 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/HasExternalIdCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/HasExternalIdCrud.java @@ -2,7 +2,7 @@ import java.util.List; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -32,7 +32,7 @@ public interface HasExternalIdCrud extends AbstractCrud */ default T retrieveByExternalId(int externalId) { Logger log = LoggerFactory.getLogger(HasExternalIdCrud.class); - log.info(getClass().getSimpleName() + " retrieving for externalId " + externalId); + log.info("{} retrieving for externalId {}", getClass().getSimpleName(), externalId); ZiggyQuery query = createZiggyQuery(componentClass()); query.column(getExternalIdFieldName()).in(externalId); @@ -45,7 +45,7 @@ default T retrieveByExternalId(int externalId) { */ default T retrieveByMaxExternalId() { Logger log = LoggerFactory.getLogger(HasExternalIdCrud.class); - log.info(getClass().getSimpleName() + " retrieving for max externalId"); + log.info("{} retrieving for max externalId", getClass().getSimpleName()); ZiggyQuery query = createZiggyQuery(componentClass()); query.column(getExternalIdFieldName()).in(retrieveMaxExternalId()); return finalize(uniqueResult(query)); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperations.java index fd6b512..38c2a66 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperations.java @@ -35,18 +35,6 @@ public class ParametersOperations extends DatabaseOperations { private PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); - public void save(ParameterSet parameterSet) { - performTransaction(() -> parameterSetCrud.persist(parameterSet)); - } - - public ParameterSet rename(ParameterSet parameterSet, String newName) { - return performTransaction(() -> parameterSetCrud.rename(parameterSet, newName)); - } - - public void delete(ParameterSet parameterSet) { - performTransaction(() -> parameterSetCrud.remove(parameterSet)); - } - /** * Populates the {@link Set} of {@link ParameterSet} instances for a pipeline instance or a * pipeline instance node, using the {@link Set} of {@link String} instances that represent the @@ -108,8 +96,7 @@ public ParameterSet updateParameterSet(String parameterSetName, ParameterSet new // is -- not what we wanted to know (we want to compare to the version actually in the // database). Hence, retrieve the parameter set in one transaction and merge in another. - ParameterSet databaseParameterSet = parameterSet(parameterSetName); - return updateParameterSet(databaseParameterSet, newParameters, false); + return updateParameterSet(parameterSet(parameterSetName), newParameters, false); } /** @@ -123,8 +110,7 @@ public ParameterSet updateParameterSet(String parameterSetName, ParameterSet new */ public ParameterSet updateParameterSet(ParameterSet parameterSet, ParameterSet newParameters, boolean forceSave) { - return updateParameterSet(parameterSet, newParameters.getParameters(), - parameterSet.getDescription(), forceSave); + return updateParameterSet(parameterSet, newParameters.getParameters(), forceSave); } /** @@ -137,23 +123,11 @@ public ParameterSet updateParameterSet(ParameterSet parameterSet, ParameterSet n * returned. */ public ParameterSet updateParameterSet(ParameterSet parameterSet, Set newParameters, - String newDescription, boolean forceSave) { - - String currentDescription = parameterSet.getDescription(); - - boolean descriptionChanged = false; - if (currentDescription == null) { - if (newDescription != null) { - descriptionChanged = true; - } - } else if (!currentDescription.equals(newDescription)) { - descriptionChanged = true; - } + boolean forceSave) { if (!Parameter.identicalParameters(parameterSet.getParameters(), newParameters) - || descriptionChanged || forceSave) { + || forceSave) { parameterSet.setParameters(newParameters); - parameterSet.setDescription(newDescription); return performTransaction(() -> parameterSetCrud().merge(parameterSet)); } return parameterSet; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrud.java index 50a3ba0..d8b69fe 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrud.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.pipeline.definition.database; -import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Set; @@ -10,7 +9,6 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.crud.ZiggyQuery; -import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions; @@ -59,41 +57,6 @@ public List retrievePipelineDefinitionNamesInUse() { return list(query); } - public void deletePipeline(PipelineDefinition pipelineDefinition) { - if (pipelineDefinition.getVersion() > 0 || pipelineDefinition.isLocked()) { - throw new PipelineException("Unable to remove " + componentNameForExceptionMessages() - + " as " + pipelineDefinition.getName() - + " is locked or its version is greater than zero"); - } - - // Must delete the nodes before deleting the pipeline because the cascade rules do not - // include delete (having Cascade.ALL would cause errors in the console when manually - // deleting individual nodes). - deleteNodes(pipelineDefinition.getRootNodes()); - remove(pipelineDefinition); - } - - /** - * Delete all of the nodes in a pipeline and clear the rootNodes List. - */ - public void deleteAllPipelineNodes(PipelineDefinition pipelineDefinition) { - List rootNodes = pipelineDefinition.getRootNodes(); - deleteNodes(rootNodes); - pipelineDefinition.setRootNodes(Collections.emptyList()); - } - - /** - * Recursively delete all of the nodes in a pipeline. - * - * @param rootNodes - */ - private void deleteNodes(List nodes) { - for (PipelineDefinitionNode node : nodes) { - deleteNodes(node.getNextNodes()); - remove(node); - } - } - public boolean processingModeExistsInDatabase(String pipelineName) { return uniqueResult(createZiggyQuery(PipelineDefinitionProcessingOptions.class) .column(PipelineDefinitionProcessingOptions_.pipelineName) diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionNodeOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionNodeOperations.java index 65921b9..d539d5b 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionNodeOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionNodeOperations.java @@ -112,10 +112,6 @@ public Set parameterSetNames(PipelineDefinitionNode pipelineDefinitionNo () -> pipelineDefinitionNodeCrud().retrieveParameterSetNames(pipelineDefinitionNode)); } - public void delete(PipelineDefinitionNode pipelineNode) { - performTransaction(() -> pipelineDefinitionCrud().remove(pipelineNode)); - } - public Set inputDataFileTypes(PipelineDefinitionNode pipelineDefinitionNode) { return performTransaction( () -> pipelineDefinitionNodeCrud().retrieveInputDataFileTypes(pipelineDefinitionNode)); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionOperations.java index b62361d..84269bd 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionOperations.java @@ -116,14 +116,6 @@ public PipelineDefinition merge(PipelineDefinition pipeline) { return performTransaction(() -> pipelineDefinitionCrud().merge(pipeline)); } - public PipelineDefinition rename(PipelineDefinition pipeline, String newName) { - return performTransaction(() -> pipelineDefinitionCrud().rename(pipeline, newName)); - } - - public void delete(PipelineDefinition pipeline) { - performTransaction(() -> pipelineDefinitionCrud().deletePipeline(pipeline)); - } - public void lock(PipelineDefinition pipelineDefinition) { performTransaction(() -> { PipelineDefinition databaseDefinition = pipelineDefinitionCrud() diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceCrud.java index 837c3f9..a70919e 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceCrud.java @@ -62,7 +62,7 @@ public List retrieve(PipelineInstanceFilter filter) { public List retrieve(Date startDate, Date endDate) { ZiggyQuery query = createZiggyQuery( PipelineInstance.class); - query.column(PipelineInstance_.startProcessingTime).between(startDate, endDate); + query.column(PipelineInstance_.created).between(startDate, endDate); return list(query); } @@ -107,7 +107,7 @@ public void updateName(long id, String newName) { query.where(builder.in(root.get("id"), Set.of(id))).set(root.get("name"), newName); int rowsUpdated = executeUpdate(query); - log.info("Updated instance name, rowsUpdated=" + rowsUpdated); + log.info("Updated instance name, rowsUpdated={}", rowsUpdated); } /** @@ -140,13 +140,13 @@ private ZiggyQuery queryForFilter( Date startTime = new Date( System.currentTimeMillis() - filter.getAgeDays() * MILLIS_PER_DAY); query.where(query.getBuilder() - .greaterThan(query.getRoot().get("startProcessingTime"), startTime)); + .greaterThan(query.getRoot().get(PipelineInstance_.created), startTime)); } // If the user wants to filter by pipeline instance name, apply that now. if (!StringUtils.isBlank(filter.getNameContains())) { - query.where( - query.getBuilder().like(query.getRoot().get("name"), filter.getNameContains())); + query.where(query.getBuilder() + .like(query.getRoot().get(PipelineInstance_.name), filter.getNameContains())); } query.column(PipelineInstance_.id).ascendingOrder(); return query; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceFilter.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceFilter.java index 33fb5f2..e3f110e 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceFilter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceFilter.java @@ -29,8 +29,8 @@ public class PipelineInstanceFilter { private Set states; /** - * Pass if PipelineInstance.startProcessingTime is within ageDays days of the time the query is - * ran. If 0, startProcessingTime is not included in the where clause. + * Pass if "PipelineInstance.created" is within ageDays days of the time the query is ran. If 0, + * "created" is not included in the where clause. */ private int ageDays = DEFAULT_AGE; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperations.java index 24d067f..c46f29d 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperations.java @@ -32,6 +32,7 @@ public class PipelineInstanceNodeOperations extends DatabaseOperations { private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); private PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); private ParametersOperations parametersOperations = new ParametersOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); /** returns the next pipeline instance nodes for execution. */ public List nextPipelineInstanceNodes(long pipelineInstanceNodeId) { @@ -84,13 +85,10 @@ public void markInstanceNodeTransitionIncomplete(long pipelineInstanceNodeId) { public PipelineInstanceNodeInformation pipelineInstanceNodeInformation( long pipelineInstanceNodeId) { - return performTransaction(() -> { - PipelineInstanceNode instanceNode = pipelineInstanceNodeCrud() - .retrieve(pipelineInstanceNodeId); - TaskCounts taskCounts = new TaskCounts( - pipelineTaskCrud().retrieveTasksForInstanceNode(instanceNode)); - return new PipelineInstanceNodeInformation(instanceNode, taskCounts); - }); + PipelineInstanceNode instanceNode = performTransaction( + () -> pipelineInstanceNodeCrud().retrieve(pipelineInstanceNodeId)); + TaskCounts taskCounts = pipelineTaskDisplayDataOperations().taskCounts(instanceNode); + return new PipelineInstanceNodeInformation(instanceNode, taskCounts); } public PipelineInstanceNode bindParameterSets(PipelineDefinitionNode definitionNode, @@ -116,26 +114,6 @@ public void addPipelineTasks(PipelineInstanceNode pipelineInstanceNode, }); } - public List distinctSoftwareRevisions(PipelineInstanceNode pipelineInstanceNode) { - return performTransaction( - () -> pipelineTaskCrud().distinctSoftwareRevisions(pipelineInstanceNode)); - } - - /** - * Returns a {@link TaskCounts} instance for a given {@link PipelineInstanceNode}. - */ - public TaskCounts taskCounts(PipelineInstanceNode pipelineInstanceNode) { - return performTransaction(() -> new TaskCounts( - pipelineTaskCrud().retrieveTasksForInstanceNode(pipelineInstanceNode))); - } - - /** - * Returns a {@link TaskCounts} instance for the given list of {@link PipelineInstanceNode}s. - */ - public TaskCounts taskCounts(List pipelineInstanceNodes) { - return performTransaction(() -> new TaskCounts(pipelineTasks(pipelineInstanceNodes))); - } - public PipelineInstance pipelineInstance(PipelineInstanceNode pipelineInstanceNode) { return performTransaction( () -> pipelineInstanceNodeCrud().retrievePipelineInstance(pipelineInstanceNode)); @@ -180,6 +158,10 @@ PipelineTaskCrud pipelineTaskCrud() { return pipelineTaskCrud; } + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud() { return pipelineModuleDefinitionCrud; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperations.java index 17016a7..417f1e2 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperations.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.pipeline.definition.database; -import java.util.ArrayList; import java.util.List; import java.util.Set; @@ -24,6 +23,7 @@ public class PipelineInstanceOperations extends DatabaseOperations { private PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); private PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); private PipelineInstanceNodeCrud pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); private PipelineDefinitionCrud pipelineDefinitionCrud = new PipelineDefinitionCrud(); private ParametersOperations parametersOperations = new ParametersOperations(); @@ -46,11 +46,6 @@ public void setInstanceToErrorsStalledState(PipelineInstance pipelineInstance) { }); } - public List distinctSoftwareRevisions(PipelineInstance pipelineInstance) { - return performTransaction( - () -> pipelineTaskCrud().distinctSoftwareRevisions(pipelineInstance)); - } - /** Merges a pipeline instance and returns the merged instance. */ public PipelineInstance merge(PipelineInstance instance) { return performTransaction(() -> pipelineInstanceCrud().merge(instance)); @@ -80,42 +75,12 @@ public List rootNodes(PipelineInstance pipelineInstance) { return performTransaction(() -> pipelineInstanceCrud().retrieveRootNodes(pipelineInstance)); } - /** - * Updates all of the {@link PipelineTask} instances associated with a particular - * {@link PipelineInstance} and returns them as a List. - */ - public List updateJobs(PipelineInstance pipelineInstance) { - List tasks = performTransaction( - () -> pipelineTaskCrud().retrieveTasksForInstance(pipelineInstance)); - List updatedTasks = new ArrayList<>(); - for (PipelineTask task : tasks) { - updatedTasks.add(pipelineTaskOperations().updateJobs(task)); - } - return updatedTasks; - } - /** * Returns a {@link TaskCounts} instance for a given {@link PipelineInstance}. */ public TaskCounts taskCounts(PipelineInstance pipelineInstance) { - return performTransaction( - () -> new TaskCounts(pipelineTaskCrud().retrieveTasksForInstance(pipelineInstance))); - } - - /** - * Determines whether all pipeline instances have completed execution. This means that all nodes - * have tasks that submitted and all tasks for all nodes are either complete or errored. - */ - public boolean allInstanceNodesExecutionComplete(PipelineInstance pipelineInstance) { - List instanceNodes = new PipelineInstanceNodeCrud() - .retrieveAll(pipelineInstance); - for (PipelineInstanceNode node : instanceNodes) { - if (!new TaskCounts(pipelineTaskCrud().retrieveTasksForInstanceNode(node)) - .isPipelineTasksExecutionComplete()) { - return false; - } - } - return true; + return performTransaction(() -> new TaskCounts( + pipelineTaskDisplayDataOperations().pipelineTaskDisplayData(pipelineInstance))); } public void addRootNode(PipelineInstance instance, PipelineInstanceNode pipelineInstanceNode) { @@ -186,6 +151,10 @@ PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + PipelineDefinitionCrud pipelineDefinitionCrud() { return pipelineDefinitionCrud; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineModuleDefinitionOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineModuleDefinitionOperations.java index 1c30892..524885c 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineModuleDefinitionOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineModuleDefinitionOperations.java @@ -41,14 +41,6 @@ public PipelineModuleDefinition merge(PipelineModuleDefinition module) { return performTransaction(() -> pipelineModuleDefinitionCrud().merge(module)); } - public PipelineModuleDefinition rename(PipelineModuleDefinition module, String newName) { - return performTransaction(() -> pipelineModuleDefinitionCrud().rename(module, newName)); - } - - public void delete(PipelineModuleDefinition module) { - performTransaction(() -> pipelineModuleDefinitionCrud().remove(module)); - } - public PipelineModuleExecutionResources pipelineModuleExecutionResources( PipelineModuleDefinition module) { return performTransaction( diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrud.java index 00c2983..5fb796d 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrud.java @@ -18,11 +18,7 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance_; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; import gov.nasa.ziggy.pipeline.definition.PipelineTask_; -import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.RemoteJob; -import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; import gov.nasa.ziggy.services.database.DatabaseService; import jakarta.persistence.criteria.Join; import jakarta.persistence.metamodel.SetAttribute; @@ -54,27 +50,6 @@ public PipelineTask retrieve(long id) { return uniqueResult(createZiggyQuery(PipelineTask.class).column(PipelineTask_.id).in(id)); } - public Set retrieveRemoteJobs(long id) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class, - RemoteJob.class); - query.column(PipelineTask_.id).in(id); - return new HashSet<>(list(query.column(PipelineTask_.remoteJobs).select())); - } - - public List retrieveExecLogs(long id) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class, - TaskExecutionLog.class); - query.column(PipelineTask_.id).in(id); - return list(query.column(PipelineTask_.execLog).select()); - } - - public List retrievePipelineTaskMetrics(long id) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class, - PipelineTaskMetrics.class); - query.column(PipelineTask_.id).in(id); - return list(query.column(PipelineTask_.summaryMetrics).select()); - } - /** * Retrieve all {@link PipelineTask}s for the specified {@link PipelineInstance} */ @@ -156,43 +131,13 @@ public List retrieveTasksForModuleAndInstance(String moduleName, /** * Retrieve all {@link PipelineTask}s that have a specific {@link PipelineDefinitionNode}. */ - public List retrieveIdsForPipelineDefinitionNode( + public List retrieveTasksForPipelineDefinitionNode( PipelineDefinitionNode pipelineDefinitionNode) { - ZiggyQuery query = createZiggyQuery(PipelineInstanceNode.class, - Long.class); + ZiggyQuery query = createZiggyQuery( + PipelineInstanceNode.class, PipelineTask.class); query.column(PipelineInstanceNode_.pipelineDefinitionNode).in(pipelineDefinitionNode); - Join taskJoin = query - .column(PipelineInstanceNode_.pipelineTasks) - .join(); - query.select(taskJoin.get(PipelineTask_.id)); - return list(query); - } - - /** - * Retrieves all {@link PipelineTask}s for the specified {@link PipelineInstance} and the - * specified {@link ProcessingStep}s. - */ - public List retrieveAll(PipelineInstance pipelineInstance, - Set processingSteps) { - - ZiggyQuery query = createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.pipelineInstanceId).in(pipelineInstance.getId()); - query.column(PipelineTask_.processingStep).in(processingSteps); - query.column(PipelineTask_.id).ascendingOrder(); - - return list(query); - } - - /** - * Retrieve all {@link PipelineTask}s for the specified {@link PipelineInstance} with errors. - */ - public List retrieveErroredTasks(PipelineInstance instance) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.pipelineInstanceId).in(instance.getId()); - query.column(PipelineTask_.error).in(true); - query.column(PipelineTask_.id).ascendingOrder(); - + query.column(PipelineInstanceNode_.pipelineTasks).select(); return list(query); } @@ -206,13 +151,13 @@ public List retrieveAll(Collection pipelineTaskIds) { if (pipelineTaskIds.isEmpty()) { return new ArrayList<>(); } - ZiggyQuery query = createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.id).chunkedIn(pipelineTaskIds); - query.column(PipelineTask_.id).ascendingOrder(); - return list(query); + return chunkedQuery(new ArrayList<>(pipelineTaskIds), + chunk -> list(createZiggyQuery(PipelineTask.class).column(PipelineTask_.id) + .ascendingOrder() + .in(chunk))); } - private List taskIdsForPipelineInstanceNode(PipelineInstanceNode node) { + List taskIdsForPipelineInstanceNode(PipelineInstanceNode node) { return taskIdsForPipelineInstanceNode(node.getId()); } @@ -227,45 +172,6 @@ private List taskIdsForPipelineInstanceNode(long nodeId) { return list(taskIdsQuery); } - /** - * Retrieve the list of distinct softwareRevisions for the specified node. Used for reporting - */ - public List distinctSoftwareRevisions(PipelineInstanceNode node) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class, String.class); - query.column(PipelineTask_.id).in(taskIdsForPipelineInstanceNode(node)); - query.column(PipelineTask_.softwareRevision).select(); - query.column(PipelineTask_.softwareRevision).ascendingOrder(); - query.distinct(true); - - return list(query); - } - - /** - * Retrieve the list of distinct softwareRevisions for the specified pipeline instance. Used for - * reporting - */ - public List distinctSoftwareRevisions(PipelineInstance instance) { - ZiggyQuery query = createZiggyQuery(PipelineTask.class, String.class); - query.column(PipelineTask_.pipelineInstanceId).in(instance.getId()); - query.column(PipelineTask_.softwareRevision).select(); - query.column(PipelineTask_.softwareRevision).ascendingOrder(); - query.distinct(true); - - return list(query); - } - - /** - * Locates tasks that have "stale" states, i.e., tasks that were in process when the cluster was - * shut down. - */ - public List retrieveTasksWithStaleStates() { - ZiggyQuery query = createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.processingStep).in(ProcessingStep.processingSteps()); - query.column(PipelineTask_.error).in(false); - query.distinct(true); - return list(query); - } - @Override public Class componentClass() { return PipelineTask.class; @@ -343,6 +249,10 @@ public Set retrieveModelTypes(PipelineTask pipelineTask) { pipelineTask); } + public PipelineInstance retrievePipelineInstance(long pipelineTaskId) { + return retrievePipelineInstance(retrieve(pipelineTaskId)); + } + public PipelineInstance retrievePipelineInstance(PipelineTask pipelineTask) { ZiggyQuery query = createZiggyQuery( PipelineInstance.class); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataCrud.java new file mode 100644 index 0000000..60dbebb --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataCrud.java @@ -0,0 +1,222 @@ +package gov.nasa.ziggy.pipeline.definition.database; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; +import java.util.stream.Collectors; + +import gov.nasa.ziggy.crud.AbstractCrud; +import gov.nasa.ziggy.crud.ZiggyQuery; +import gov.nasa.ziggy.pipeline.definition.ExecutionClock; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData_; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTask_; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; +import jakarta.persistence.metamodel.SingularAttribute; + +public class PipelineTaskDataCrud extends AbstractCrud { + + private final PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); + + /** For use by PipelineTaskDataOperations only. */ + PipelineTaskData retrievePipelineTaskData(PipelineTask pipelineTask) { + return retrievePipelineTaskData(List.of(pipelineTask)).get(0); + } + + /** For use by PipelineTaskDataOperations only. */ + PipelineTaskData retrievePipelineTaskData(long taskId) { + return retrievePipelineTaskData(pipelineTaskCrud().retrieve(taskId)); + } + + /** For use by PipelineTaskDataOperations only. */ + List retrievePipelineTaskData(Collection pipelineTasks) { + Set taskIds = new HashSet<>( + pipelineTasks.stream().map(PipelineTask::getId).collect(Collectors.toSet())); + + List pipelineTaskData = list( + createZiggyQuery(PipelineTaskData.class).column(PipelineTaskData_.pipelineTaskId) + .in(taskIds) + .ascendingOrder()); + + // Create objects if not already present in database. + Set newTasks = new HashSet<>(pipelineTasks); + newTasks.removeAll(pipelineTaskData.stream() + .map(PipelineTaskData::getPipelineTask) + .collect(Collectors.toSet())); + if (newTasks.size() > 0) { + List newPipelineTaskData = new ArrayList<>(); + for (PipelineTask pipelineTask : newTasks) { + newPipelineTaskData.add(new PipelineTaskData(pipelineTask)); + } + persist(newPipelineTaskData); + newPipelineTaskData.addAll(pipelineTaskData); + Collections.sort(newPipelineTaskData); + return newPipelineTaskData; + } + + return pipelineTaskData; + } + + ProcessingStep retrieveProcessingStep(PipelineTask pipelineTask) { + return retrievePipelineTaskData(pipelineTask).getProcessingStep(); + } + + ExecutionClock retrieveExecutionClock(PipelineTask pipelineTask) { + return retrievePipelineTaskData(pipelineTask).getExecutionClock(); + } + + /** + * Retrieve all {@link PipelineTask}s for the specified {@link PipelineInstance} with errors. + */ + List retrieveErroredTasks(PipelineInstance pipelineInstance) { + ZiggyQuery query = createZiggyQuery(PipelineTask.class) + .column(PipelineTask_.pipelineInstanceId) + .in(pipelineInstance.getId()); + ZiggyQuery taskIdQuery = query + .ziggySubquery(PipelineTaskData.class, Long.class) + .column(PipelineTaskData_.pipelineTaskId) + .select() + .column(PipelineTaskData_.error) + .in(true); + query.column(PipelineTask_.id).in(taskIdQuery); + return list(query); + } + + /** + * Retrieves the tasks for all {@link PipelineTask}s in the specified {@link PipelineInstance} + * for the specified {@link ProcessingStep}s. + */ + List retrievePipelineTasks(PipelineInstance pipelineInstance, + Set processingSteps) { + + ZiggyQuery query = createZiggyQuery(PipelineTask.class, Long.class) + .column(PipelineTask_.pipelineInstanceId) + .in(pipelineInstance.getId()); + ZiggyQuery taskIdQuery = query + .ziggySubquery(PipelineTaskData.class, Long.class) + .column(PipelineTaskData_.pipelineTaskId) + .select() + .column(PipelineTaskData_.processingStep) + .in(processingSteps); + query.column(PipelineTask_.id).select().in(taskIdQuery); + + // TODO When PipelineTaskData contains a PipelineTask, this query goes away + return list(query).stream() + .map(t -> pipelineTaskCrud().retrieve(t)) + .collect(Collectors.toList()); + } + + /** + * Locates tasks that have "stale" states, i.e., tasks that were in process when the cluster was + * shut down. + */ + List retrieveTasksWithStaleStates() { + return list(createZiggyQuery(PipelineTaskData.class, Long.class) + .column(PipelineTaskData_.pipelineTaskId) + .select() + .column(PipelineTaskData_.processingStep) + .in(ProcessingStep.processingSteps()) + .column(PipelineTaskData_.error) + .in(false) + .distinct(true)); + } + + /** + * Retrieves the list of distinct softwareRevisions for the specified node. + */ + List distinctSoftwareRevisions(PipelineInstanceNode pipelineInstanceNode) { + List distinctSoftwareRevisions = new ArrayList<>(); + + distinctSoftwareRevisions.addAll(list( + softwareRevisionQuery(pipelineInstanceNode, PipelineTaskData_.ziggySoftwareRevision))); + distinctSoftwareRevisions.addAll(list(softwareRevisionQuery(pipelineInstanceNode, + PipelineTaskData_.pipelineSoftwareRevision))); + return distinctSoftwareRevisions; + } + + private ZiggyQuery softwareRevisionQuery( + PipelineInstanceNode pipelineInstanceNode, + SingularAttribute versionType) { + return createZiggyQuery(PipelineTaskData.class, String.class) + .column(PipelineTaskData_.pipelineTaskId) + .in(pipelineTaskCrud().taskIdsForPipelineInstanceNode(pipelineInstanceNode)) + .column(versionType) + .select() + .column(versionType) + .ascendingOrder() + .distinct(true); + } + + /** + * Retrieves the list of distinct softwareRevisions for the specified pipeline instance. + */ + List distinctSoftwareRevisions(PipelineInstance pipelineInstance) { + List distinctSoftwareRevisions = new ArrayList<>(); + distinctSoftwareRevisions.addAll( + list(softwareRevisionQuery(pipelineInstance, PipelineTaskData_.ziggySoftwareRevision))); + distinctSoftwareRevisions.addAll(list( + softwareRevisionQuery(pipelineInstance, PipelineTaskData_.pipelineSoftwareRevision))); + return distinctSoftwareRevisions; + } + + private ZiggyQuery softwareRevisionQuery( + PipelineInstance pipelineInstance, + SingularAttribute versionType) { + ZiggyQuery softwareRevisionQuery = createZiggyQuery( + PipelineTaskData.class, String.class).column(versionType) + .select() + .ascendingOrder() + .distinct(true); + ZiggyQuery taskIdQuery = softwareRevisionQuery + .ziggySubquery(PipelineTask.class, Long.class) + .column(PipelineTask_.pipelineInstanceId) + .in(pipelineInstance.getId()) + .column(PipelineTask_.id) + .select(); + softwareRevisionQuery.column(PipelineTaskData_.pipelineTaskId).in(taskIdQuery); + return softwareRevisionQuery; + } + + public List retrievePipelineTaskMetrics(PipelineTask pipelineTask) { + return list(createZiggyQuery(PipelineTaskData.class, PipelineTaskMetric.class) + .column(PipelineTaskData_.pipelineTaskId) + .in(pipelineTask.getId()) + .column(PipelineTaskData_.pipelineTaskMetrics) + .select()); + } + + public List retrieveTaskExecutionLogs(PipelineTask pipelineTask) { + return list(createZiggyQuery(PipelineTaskData.class, TaskExecutionLog.class) + .column(PipelineTaskData_.pipelineTaskId) + .in(pipelineTask.getId()) + .column(PipelineTaskData_.taskExecutionLogs) + .select()); + } + + public Set retrieveRemoteJobs(PipelineTask pipelineTask) { + return new TreeSet<>(list(createZiggyQuery(PipelineTaskData.class, RemoteJob.class) + .column(PipelineTaskData_.pipelineTaskId) + .in(pipelineTask.getId()) + .column(PipelineTaskData_.remoteJobs) + .select())); + } + + @Override + public Class componentClass() { + return PipelineTaskData.class; + } + + PipelineTaskCrud pipelineTaskCrud() { + return pipelineTaskCrud; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperations.java new file mode 100644 index 0000000..2ab5ef1 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperations.java @@ -0,0 +1,555 @@ +package gov.nasa.ziggy.pipeline.definition.database; + +import java.text.MessageFormat; +import java.util.Collection; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.apache.commons.collections4.CollectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.module.remote.QueueCommandManager; +import gov.nasa.ziggy.module.remote.RemoteJobInformation; +import gov.nasa.ziggy.pipeline.definition.ExecutionClock; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.pipeline.definition.RemoteJob.RemoteJobQstatInfo; +import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; +import gov.nasa.ziggy.services.database.DatabaseOperations; + +/** + * {@link DatabaseOperations} class to access fields from the {@link PipelineTaskData} table. + * + * @author PT + * @author Bill Wohler + */ +public class PipelineTaskDataOperations extends DatabaseOperations { + + private static final Logger log = LoggerFactory.getLogger(PipelineTaskDataOperations.class); + + /** + * Log filename format. This is used by {@link MessageFormat} to produce the filename for a log + * file. The first element is the basename, "instanceId-taskId-moduleName" (i.e., + * "100-200-foo"). The second element is a job index, needed when running tasks on a remote + * system that can produce multiple log files per task (i.e., one per remote job). The final + * element is the task log index, a value that increments as a task gets executed or rerun, + * which allows the logs to be sorted into the order in which they were generated. + */ + private static final String LOG_FILENAME_FORMAT = "{0}.{1}-{2}.log"; + + private final PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); + private final PipelineTaskDataCrud pipelineTaskDataCrud = new PipelineTaskDataCrud(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + private final PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); + private final PipelineInstanceNodeCrud pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); + + public void createPipelineTaskData(PipelineTask pipelineTask, ProcessingStep processingStep) { + PipelineTaskData pipelineTaskData = new PipelineTaskData(pipelineTask); + if (processingStep != null) { + pipelineTaskData.setProcessingStep(processingStep); + } + performTransaction(() -> pipelineTaskDataCrud().persist(pipelineTaskData)); + } + + public ProcessingStep processingStep(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrieveProcessingStep(pipelineTask)); + } + + public List pipelineTasks(PipelineInstance pipelineInstance, + Set processingSteps) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTasks(pipelineInstance, processingSteps)); + } + + /** + * Updates the processing step of a {@link PipelineTask} and simultaneously starts or stops + * execution clocks as needed and updates the state of that task's {@link PipelineInstance} if + * need be. + */ + public void updateProcessingStep(PipelineTask pipelineTask, ProcessingStep processingStep) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setProcessingStep(processingStep); + if (processingStep == ProcessingStep.INITIALIZING + || processingStep == ProcessingStep.COMPLETE) { + pipelineTaskData.getExecutionClock().stop(); + } else { + pipelineTaskData.getExecutionClock().start(); + } + }); + updateInstanceState(pipelineTask); + } + + public boolean hasErrored(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask).isError()); + } + + /** + * Sets the error flag on the pipeline task and and simultaneously stops execution clock and + * updates the state of that task's {@link PipelineInstance} if need be. + */ + public void taskErrored(PipelineTask pipelineTask) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setError(true); + pipelineTaskData.getExecutionClock().stop(); + }); + updateInstanceState(pipelineTask); + } + + public void clearError(PipelineTask pipelineTask) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setError(false); + }); + } + + /** + * For testing only. Use {@link #taskErrored(PipelineTask)} or {@link #clearError(PipelineTask)} + * instead. + */ + public void setError(PipelineTask pipelineTask, boolean error) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setError(error); + }); + } + + public boolean haltRequested(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask).isHaltRequested()); + } + + public void setHaltRequested(PipelineTask pipelineTask, boolean haltRequested) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setHaltRequested(haltRequested); + }); + } + + private PipelineInstance updateInstanceState(PipelineTask pipelineTask) { + + return performTransaction(() -> { + PipelineInstance pipelineInstance = pipelineTaskCrud() + .retrievePipelineInstance(pipelineTask); + + TaskCounts instanceNodeCounts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskCrud().retrievePipelineInstanceNode(pipelineTask)); + + PipelineInstance.State state = PipelineInstance.State.INITIALIZED; + if (allInstanceNodesExecutionComplete(pipelineInstance)) { + + // If all the instance nodes are done, we can set the instance state to + // either completed or stalled. + state = instanceNodeCounts.isPipelineTasksComplete() + ? PipelineInstance.State.COMPLETED + : PipelineInstance.State.ERRORS_STALLED; + } else if (instanceNodeCounts.isPipelineTasksExecutionComplete()) { + + // If the current node is done, then the state is either stalled or processing. + state = instanceNodeCounts.isPipelineTasksComplete() + ? PipelineInstance.State.PROCESSING + : PipelineInstance.State.ERRORS_STALLED; + } else { + + // If the current instance node is still grinding away, then the state is either + // errors running or processing + state = instanceNodeCounts.getTotalCounts().getFailedTaskCount() == 0 + ? PipelineInstance.State.PROCESSING + : PipelineInstance.State.ERRORS_RUNNING; + } + + state.setExecutionClockState(pipelineInstance); + pipelineInstance.setState(state); + + return pipelineInstanceCrud().merge(pipelineInstance); + }); + } + + /** + * Determines whether all pipeline instances have completed execution. This means that all nodes + * have tasks that submitted and all tasks for all nodes are either complete or errored. + */ + private boolean allInstanceNodesExecutionComplete(PipelineInstance pipelineInstance) { + List instanceNodes = pipelineInstanceNodeCrud() + .retrieveAll(pipelineInstance); + for (PipelineInstanceNode node : instanceNodes) { + if (!pipelineTaskDisplayDataOperations().taskCounts(node) + .isPipelineTasksExecutionComplete()) { + return false; + } + } + return true; + } + + public List erroredPipelineTasks(PipelineInstance pipelineInstance) { + return performTransaction( + () -> pipelineTaskDataCrud().retrieveErroredTasks(pipelineInstance)); + } + + public ExecutionClock executionClock(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrieveExecutionClock(pipelineTask)); + } + + public void updateWorkerInfo(PipelineTask pipelineTask, String workerHost, int workerThread) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setWorkerHost(workerHost); + pipelineTaskData.setWorkerThread(workerThread); + }); + } + + public List distinctSoftwareRevisions(PipelineInstance pipelineInstance) { + return performTransaction( + () -> pipelineTaskDataCrud().distinctSoftwareRevisions(pipelineInstance)); + } + + public List distinctSoftwareRevisions(PipelineInstanceNode pipelineInstanceNode) { + return performTransaction( + () -> pipelineTaskDataCrud().distinctSoftwareRevisions(pipelineInstanceNode)); + } + + public void updateZiggySoftwareRevision(PipelineTask pipelineTask, String softwareRevision) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setZiggySoftwareRevision(softwareRevision); + }); + } + + public void updatePipelineSoftwareRevision(PipelineTask pipelineTask, String softwareRevision) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setPipelineSoftwareRevision(softwareRevision); + }); + } + + public int autoResubmitCount(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask) + .getAutoResubmitCount()); + } + + public boolean retrying(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask).isRetry()); + } + + public void prepareTasksForManualResubmit(Collection pipelineTasks) { + performTransaction(() -> { + List pipelineTaskDataList = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTasks); + for (PipelineTaskData pipelineTaskData : pipelineTaskDataList) { + pipelineTaskData.resetAutoResubmitCount(); + pipelineTaskData.setRetry(true); + } + }); + } + + public void prepareTaskForAutoResubmit(PipelineTask pipelineTask) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.incrementAutoResubmitCount(); + taskErrored(pipelineTask); + }); + } + + public void prepareTaskForRestart(PipelineTask pipelineTask) { + boolean taskErrored = hasErrored(pipelineTask); + ProcessingStep processingStep = processingStep(pipelineTask); + + if (!taskErrored && (processingStep != ProcessingStep.COMPLETE + || failedSubtaskCount(pipelineTask) <= 0)) { + log.warn("Task {} is on step {} without errors, not restarting", pipelineTask, + processingStep); + return; + } + + log.info("Restarting task {} on step {} {} errors", pipelineTask, processingStep, + taskErrored ? "with" : "without"); + + clearError(pipelineTask); + } + + private int failedSubtaskCount(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask) + .getFailedSubtaskCount()); + } + + public SubtaskCounts subtaskCounts(PipelineTask pipelineTask) { + PipelineTaskData pipelineTaskData = performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask)); + return new SubtaskCounts(pipelineTaskData.getTotalSubtaskCount(), + pipelineTaskData.getCompletedSubtaskCount(), pipelineTaskData.getFailedSubtaskCount()); + } + + /** + * Updates the current subtask counts. Negative counts are ignored and are not transmitted to + * the database. + */ + public void updateSubtaskCounts(PipelineTask pipelineTask, int totalSubtaskCount, + int completedSubtaskCount, int failedSubtaskCount) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + if (totalSubtaskCount >= 0) { + pipelineTaskData.setTotalSubtaskCount(totalSubtaskCount); + } + if (completedSubtaskCount >= 0) { + pipelineTaskData.setCompletedSubtaskCount(completedSubtaskCount); + } + if (failedSubtaskCount >= 0) { + pipelineTaskData.setFailedSubtaskCount(failedSubtaskCount); + } + }); + } + + public void incrementFailureCount(PipelineTask pipelineTask) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.incrementFailureCount(); + }); + } + + public AlgorithmType algorithmType(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskData(pipelineTask).getAlgorithmType()); + } + + /** Sets the algorithm type for the given pipeline task. The task log index is incremented. */ + public void updateAlgorithmType(PipelineTask pipelineTask, AlgorithmType algorithmType) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setAlgorithmType(algorithmType); + pipelineTaskData.incrementTaskLogIndex(); + }); + } + + public void incrementTaskLogIndex(PipelineTask pipelineTask) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.incrementTaskLogIndex(); + }); + } + + /** + * Produces a file name for log files of the following form: {@code + * --.-.log + * } + * + * @param jobIndex index for the current job. Each pipeline task's algorithm execution can be + * performed across multiple independent jobs; this index identifies a specific job out of the + * set that are running for the current task. + */ + public String logFilename(PipelineTask pipelineTask, int jobIndex) { + return performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + return logFilename(pipelineTask, jobIndex, pipelineTaskData.getTaskLogIndex()); + }); + } + + /** + * Produces a file name for log files like {@link #logFilename(PipelineTask, int)}, except the + * taskLogIndex is given rather than obtained from the pipeline task. Good for testing without a + * database. + */ + public String logFilename(PipelineTask pipelineTask, int jobIndex, int taskLogIndex) { + return MessageFormat.format(LOG_FILENAME_FORMAT, pipelineTask.taskBaseName(), jobIndex, + taskLogIndex); + } + + public List pipelineTaskMetrics(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrievePipelineTaskMetrics(pipelineTask)); + } + + public Map> taskMetricsByTask( + PipelineInstanceNode node) { + return performTransaction(() -> { + HashMap> taskMetricsByTask = new HashMap<>(); + for (PipelineTask pipelineTask : pipelineInstanceNodeCrud() + .retrievePipelineTasks(List.of(node))) { + taskMetricsByTask.put(pipelineTask, pipelineTaskMetrics(pipelineTask)); + } + return taskMetricsByTask; + }); + } + + public void updatePipelineTaskMetrics(PipelineTask pipelineTask, + List pipelineTaskMetrics) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setPipelineTaskMetrics(pipelineTaskMetrics); + }); + } + + public List taskExecutionLogs(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineTaskDataCrud().retrieveTaskExecutionLogs(pipelineTask)); + } + + /** + * Creates and saves a new {@link TaskExecutionLog} instance. + */ + public void addTaskExecutionLog(PipelineTask pipelineTask, String workerHost, int workerThread, + long startProcessingTimeMillis) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + TaskExecutionLog taskExecutionLog = new TaskExecutionLog(workerHost, workerThread); + taskExecutionLog.setStartProcessingTime(new Date(startProcessingTimeMillis)); + taskExecutionLog.setInitialProcessingStep(pipelineTaskData.getProcessingStep()); + pipelineTaskData.getTaskExecutionLogs().add(taskExecutionLog); + }); + } + + /** + * Updates the end processing time and final processing step fields in the last task execution + * log. + */ + public void updateLastTaskExecutionLog(PipelineTask pipelineTask, Date endProcessingTime) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + List taskExecutionLogs = pipelineTaskData.getTaskExecutionLogs(); + log.debug("taskExecutionLogs size={}", taskExecutionLogs.size()); + + if (CollectionUtils.isEmpty(taskExecutionLogs)) { + log.warn("Task execution log is missing or empty for task {}", pipelineTask); + return; + } + TaskExecutionLog currentTaskExecutionLog = taskExecutionLogs + .get(taskExecutionLogs.size() - 1); + currentTaskExecutionLog.setEndProcessingTime(endProcessingTime); + currentTaskExecutionLog.setFinalProcessingStep(pipelineTaskData.getProcessingStep()); + }); + } + + public Set remoteJobs(PipelineTask pipelineTask) { + return performTransaction(() -> pipelineTaskDataCrud().retrieveRemoteJobs(pipelineTask)); + } + + public void updateRemoteJobs(PipelineTask pipelineTask, Set remoteJobs) { + performTransaction(() -> { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + pipelineTaskData.setRemoteJobs(remoteJobs); + }); + } + + /** + * Updates all of the {@link PipelineTask} instances associated with a particular + * {@link PipelineInstance}. + */ + public void updateJobs(PipelineInstance pipelineInstance) { + List tasks = performTransaction( + () -> pipelineTaskCrud().retrieveTasksForInstance(pipelineInstance)); + for (PipelineTask task : tasks) { + updateJobs(task); + } + } + + public void updateJobs(PipelineTask pipelineTask) { + updateJobs(pipelineTask, false); + } + + /** + * Updates the state of all {@link RemoteJob}s associated with a {@link PipelineTask}. Any jobs + * that have completed since the last update will be marked as finished and get their final cost + * estimates calculated; any that are still running will get an up-to-the-minute cost estimate + * calculated. If the boolean argument is true, all jobs associated with the pipeline task will + * be moved to the finished state. This is useful when the pipeline task algorithm has + * completed, and we want to ensure that all the jobs that were used for the task are correctly + * recorded as complete. + */ + public void updateJobs(PipelineTask pipelineTask, boolean markJobsCompleted) { + QueueCommandManager queueCommandManager = queueCommandManager(); + Set remoteJobs = remoteJobs(pipelineTask); + for (RemoteJob job : remoteJobs) { + if (job.isFinished()) { + continue; + } + RemoteJobQstatInfo jobInfo = queueCommandManager.remoteJobQstatInfo(job.getJobId()); + job.setCostEstimate(jobInfo.costEstimate()); + + // Is the job finished? + if (markJobsCompleted ? true : queueCommandManager.exitStatus(job.getJobId()) != null) { + log.info("Job {} marked as finished", job.getJobId()); + job.setFinished(true); + log.info("Job {} cost estimate is {}", job.getJobId(), job.getCostEstimate()); + } else { + log.info("Incomplete job {} running cost estimate is {}", job.getJobId(), + job.getCostEstimate()); + } + } + + updateRemoteJobs(pipelineTask, remoteJobs); + } + + /** + * Creates instances of {@link RemoteJob} in the database using information obtained from the + * collection of {@link RemoteJobInformation} instances. The instances are initialized to + * estimated cost of zero and unfinished status and are added to the current set of remote jobs. + */ + public void addRemoteJobs(PipelineTask pipelineTask, + List remoteJobsInformation) { + Set remoteJobs = remoteJobs(pipelineTask); + for (RemoteJobInformation remoteJobInformation : remoteJobsInformation) { + remoteJobs.add(new RemoteJob(remoteJobInformation.getJobId())); + } + updateRemoteJobs(pipelineTask, remoteJobs); + } + + PipelineTaskCrud pipelineTaskCrud() { + return pipelineTaskCrud; + } + + PipelineTaskDataCrud pipelineTaskDataCrud() { + return pipelineTaskDataCrud; + } + + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + + PipelineInstanceCrud pipelineInstanceCrud() { + return pipelineInstanceCrud; + } + + PipelineInstanceNodeCrud pipelineInstanceNodeCrud() { + return pipelineInstanceNodeCrud; + } + + QueueCommandManager queueCommandManager() { + return QueueCommandManager.newInstance(); + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperations.java new file mode 100644 index 0000000..fca4e6e --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperations.java @@ -0,0 +1,102 @@ +package gov.nasa.ziggy.pipeline.definition.database; + +import java.util.List; +import java.util.stream.Collectors; + +import org.hibernate.Hibernate; + +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.services.database.DatabaseOperations; + +/** + * {@link DatabaseOperations} class to access fields from the {@link PipelineTaskData} table. + * + * @author PT + * @author Bill Wohler + */ + +public class PipelineTaskDisplayDataOperations extends DatabaseOperations { + private final PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); + private final PipelineTaskDataCrud pipelineTaskDataCrud = new PipelineTaskDataCrud(); + private final PipelineInstanceNodeCrud pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); + + public PipelineTaskDisplayData pipelineTaskDisplayData(PipelineTask pipelineTask) { + PipelineTaskData pipelineTaskData = performTransaction(() -> { + PipelineTaskData taskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTask); + initializeCollections(List.of(taskData)); + return taskData; + }); + return new PipelineTaskDisplayData(pipelineTaskData); + } + + public List pipelineTaskDisplayData( + PipelineInstance pipelineInstance) { + List pipelineTasks = performTransaction( + () -> pipelineTaskCrud().retrieveTasksForInstance(pipelineInstance)); + return pipelineTaskDisplayData(pipelineTasks); + } + + public List pipelineTaskDisplayData( + PipelineInstanceNode pipelineInstanceNode) { + return pipelineTaskDisplayDataForNodes(List.of(pipelineInstanceNode)); + } + + public List pipelineTaskDisplayDataForNodes( + List pipelineInstanceNodes) { + return pipelineTaskDisplayData(performTransaction( + () -> pipelineInstanceNodeCrud().retrievePipelineTasks(pipelineInstanceNodes))); + } + + public List pipelineTaskDisplayData(List pipelineTasks) { + List pipelineTaskData = performTransaction(() -> { + List taskData = pipelineTaskDataCrud() + .retrievePipelineTaskData(pipelineTasks); + initializeCollections(taskData); + return taskData; + }); + return createPipelineTaskDisplayData(pipelineTaskData); + } + + private void initializeCollections(List taskData) { + taskData.forEach(t -> Hibernate.initialize(t.getPipelineTaskMetrics())); + taskData.forEach(t -> Hibernate.initialize(t.getRemoteJobs())); + } + + private List createPipelineTaskDisplayData( + List pipelineTaskData) { + + return pipelineTaskData.stream() + .map(PipelineTaskDisplayData::new) + .collect(Collectors.toList()); + } + + public TaskCounts taskCounts(PipelineTask pipelineTask) { + return new TaskCounts(pipelineTaskDisplayData(List.of(pipelineTask))); + } + + public TaskCounts taskCounts(PipelineInstanceNode pipelineInstanceNode) { + return new TaskCounts(pipelineTaskDisplayData(pipelineInstanceNode)); + } + + public TaskCounts taskCounts(List pipelineInstanceNodes) { + return new TaskCounts(pipelineTaskDisplayDataForNodes(pipelineInstanceNodes)); + } + + PipelineTaskCrud pipelineTaskCrud() { + return pipelineTaskCrud; + } + + PipelineTaskDataCrud pipelineTaskDataCrud() { + return pipelineTaskDataCrud; + } + + PipelineInstanceNodeCrud pipelineInstanceNodeCrud() { + return pipelineInstanceNodeCrud; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperations.java index f0fdde7..0864e7c 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperations.java @@ -1,25 +1,20 @@ package gov.nasa.ziggy.pipeline.definition.database; -import static gov.nasa.ziggy.services.process.AbstractPipelineProcess.getProcessInfo; - import java.lang.reflect.InvocationTargetException; import java.util.Collection; -import java.util.Date; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; -import org.hibernate.Hibernate; -import org.jfree.util.Log; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.data.datastore.DataFileType; -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; -import gov.nasa.ziggy.module.remote.QstatMonitor; -import gov.nasa.ziggy.module.remote.QueueCommandManager; +import gov.nasa.ziggy.module.AlgorithmExecutor; +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.module.remote.RemoteExecutor; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.ModelRegistry; import gov.nasa.ziggy.pipeline.definition.ModelType; @@ -27,20 +22,13 @@ import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.RemoteJob; -import gov.nasa.ziggy.pipeline.definition.TaskCounts; -import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; import gov.nasa.ziggy.services.database.DatabaseOperations; -import gov.nasa.ziggy.services.messages.TaskRequest; -import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.worker.WorkerResources; @@ -58,6 +46,8 @@ public class PipelineTaskOperations extends DatabaseOperations { private PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); private PipelineInstanceNodeCrud pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); private PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); + private PipelineTaskDataCrud pipelineTaskDataCrud = new PipelineTaskDataCrud(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); @@ -72,9 +62,13 @@ public class PipelineTaskOperations extends DatabaseOperations { public ClearStaleStateResults clearStaleTaskStates() { return performTransaction(() -> { ClearStaleStateResults results = new ClearStaleStateResults(); - List staleTasks = pipelineTaskCrud().retrieveTasksWithStaleStates(); + List staleTaskIds = pipelineTaskDataCrud().retrieveTasksWithStaleStates(); + List staleTasks = pipelineTaskCrud().retrieveAll(staleTaskIds); results.totalUpdatedTaskCount += staleTasks.size(); for (PipelineTask task : staleTasks) { + if (pipelineTaskDataOperations().haltRequested(task)) { + continue; + } long instanceId = task.getPipelineInstanceId(); long instanceNodeId = pipelineTaskCrud().retrievePipelineInstanceNode(task).getId(); @@ -84,23 +78,23 @@ public ClearStaleStateResults clearStaleTaskStates() { // If the task was executing remotely, and it was queued or executing, we can try // to resume monitoring on it -- the jobs may have continued to run while the // supervisor was down. - if (task.getProcessingMode() != null - && task.getProcessingMode().equals(AlgorithmType.REMOTE)) { - ProcessingStep processingStep = task.getProcessingStep(); + AlgorithmType algorithmType = pipelineTaskDataOperations().algorithmType(task); + if (algorithmType != null && algorithmType == AlgorithmType.REMOTE) { + ProcessingStep processingStep = pipelineTaskDataCrud() + .retrieveProcessingStep(task); if (processingStep == ProcessingStep.QUEUED || processingStep == ProcessingStep.EXECUTING) { - log.info("Resuming monitoring for task={}", task.getId()); - TaskRequest taskRequest = new TaskRequest(instanceId, instanceNodeId, - pipelineTaskCrud().retrievePipelineDefinitionNode(task).getId(), - task.getId(), Priority.HIGHEST, false, - PipelineModule.RunMode.RESUME_MONITORING); - ZiggyMessenger.publish(taskRequest); - continue; + RemoteExecutor remoteExecutor = (RemoteExecutor) AlgorithmExecutor + .newRemoteInstance(task); + if (remoteExecutor.resumeMonitoring()) { + continue; + } } } results.uniqueInstanceIds.add(instanceId); - log.info("Setting error flag on task {}", task.getId()); - updateJobs(taskErrored(task)); + log.info("Setting error flag on task {}", task); + pipelineTaskDataOperations().taskErrored(task); + pipelineTaskDataOperations().updateJobs(task); } log.info("totalUpdatedTaskCount={} in instances={}", results.totalUpdatedTaskCount, results.uniqueInstanceIds); @@ -109,17 +103,17 @@ public ClearStaleStateResults clearStaleTaskStates() { }); } - public WorkerResources workerResourcesForTask(long taskId) { - return performTransaction(() -> pipelineDefinitionNodeCrud() - .retrieveExecutionResources(pipelineTaskCrud() - .retrievePipelineDefinitionNode(pipelineTaskCrud().retrieve(taskId))) - .workerResources()); + public WorkerResources workerResourcesForTask(PipelineTask pipelineTask) { + return performTransaction( + () -> pipelineDefinitionNodeCrud() + .retrieveExecutionResources( + pipelineTaskCrud().retrievePipelineDefinitionNode(pipelineTask)) + .workerResources()); } public PipelineDefinitionNodeExecutionResources executionResources(PipelineTask pipelineTask) { - return performTransaction(() -> pipelineDefinitionNodeCrud() - .retrieveExecutionResources(pipelineTaskCrud().retrievePipelineDefinitionNode( - pipelineTaskCrud().retrieve(pipelineTask.getId())))); + return performTransaction(() -> pipelineDefinitionNodeCrud().retrieveExecutionResources( + pipelineTaskCrud().retrievePipelineDefinitionNode(pipelineTask))); } public ModelRegistry modelRegistry(PipelineTask pipelineTask) { @@ -127,44 +121,6 @@ public ModelRegistry modelRegistry(PipelineTask pipelineTask) { () -> pipelineTaskCrud().retrievePipelineInstance(pipelineTask).getModelRegistry()); } - /** Prepares tasks for manual resubmission by resetting their auto-resubmit counts. */ - public List prepareTasksForManualResubmit(List taskIds) { - return performTransaction(() -> { - List tasksToResubmit = pipelineTaskCrud().retrieveAll(taskIds); - for (PipelineTask pipelineTask : tasksToResubmit) { - pipelineTask.resetAutoResubmitCount(); - pipelineTask.setRetry(true); - pipelineTaskCrud().merge(pipelineTask); - } - return tasksToResubmit; - }); - } - - public PipelineTask prepareTaskForAutoResubmit(PipelineTask pipelineTask) { - return performTransaction(() -> { - pipelineTask.incrementAutoResubmitCount(); - return taskErrored(pipelineTask); - }); - } - - public void setLocalExecution(long pipelineTaskId) { - performTransaction(() -> { - PipelineTask pipelineTask = pipelineTaskCrud().retrieve(pipelineTaskId); - pipelineTask.setRemoteExecution(false); - pipelineTask.incrementTaskLogIndex(); - pipelineTaskCrud().merge(pipelineTask); - }); - } - - public void setRemoteExecution(long pipelineTaskId) { - performTransaction(() -> { - PipelineTask pipelineTask = pipelineTaskCrud().retrieve(pipelineTaskId); - pipelineTask.setRemoteExecution(true); - pipelineTask.incrementTaskLogIndex(); - pipelineTaskCrud().merge(pipelineTask); - }); - } - /** * Merges a pipeline task and returns the merged task. The merge will not be attempted if the * call is made inside another transaction. @@ -188,170 +144,16 @@ public PipelineTask pipelineTask(long id) { return performTransaction(() -> pipelineTaskCrud().retrieve(id)); } - public List summaryMetrics(PipelineTask pipelineTask) { - return performTransaction( - () -> pipelineTaskCrud().retrievePipelineTaskMetrics(pipelineTask.getId())); - } - - public Map> taskMetricsByTask( - PipelineInstanceNode node) { - return performTransaction(() -> { - HashMap> taskMetricsByTask = new HashMap<>(); - for (PipelineTask task : pipelineTasks(node)) { - taskMetricsByTask.put(task, summaryMetrics(task)); - } - return taskMetricsByTask; - }); - } - - public List execLogs(PipelineTask pipelineTask) { - return performTransaction(() -> pipelineTaskCrud().retrieveExecLogs(pipelineTask.getId())); - } - - public List pipelineTasks(PipelineInstanceNode pipelineInstanceNode) { - return performTransaction( - () -> pipelineInstanceNodeCrud().retrieve(pipelineInstanceNode.getId()) - .getPipelineTasks()); - } - - public List pipelineTasks(PipelineInstance pipelineInstance, - Set processingSteps) { - return performTransaction( - () -> pipelineTaskCrud().retrieveAll(pipelineInstance, processingSteps)); - } - public List pipelineTasks(Collection pipelineTaskIds) { return performTransaction(() -> pipelineTaskCrud().retrieveAll(pipelineTaskIds)); } - public List erroredPipelineTasks(PipelineInstance pipelineInstance) { - return performTransaction(() -> pipelineTaskCrud().retrieveErroredTasks(pipelineInstance)); - } - /** * Retrieves pipeline tasks for the given instance. Fields with collections are not populated. */ public List pipelineTasks(PipelineInstance pipelineInstance) { - return pipelineTasks(pipelineInstance, false); - } - - /** - * Retrieves pipeline tasks. - * - * @param pipelineInstance retrieve tasks for the given instance - * @param initialize if true, call {@link Hibernate#initialize(Object)} to populate the - * following collections in each task: summaryMetrics, execLog, producerTaskIds - */ - public List pipelineTasks(PipelineInstance pipelineInstance, boolean initialize) { - return performTransaction(() -> { - List pipelineTasks = pipelineTaskCrud() - .retrieveTasksForInstance(pipelineInstance); - if (initialize) { - for (PipelineTask pipelineTask : pipelineTasks) { - Hibernate.initialize(pipelineTask.getSummaryMetrics()); - Hibernate.initialize(pipelineTask.getExecLog()); - } - } - return pipelineTasks; - }); - } - - /** - * Creates a new {@link TaskExecutionLog} instance and attaches it to a given - * {@link PipelineTask}. The merged pipeline task is returned. - *

- * We're doing it this way because there seems to be no way to do this if the new task execution - * log is transient and the pipeline task is detached. Thus we do it all in one transaction. - */ - public PipelineTask addTaskExecutionLog(long taskId, int workerNumber, - long startProcessingTimeMillis) { - return performTransaction(() -> { - PipelineTask pipelineTask = pipelineTaskCrud().retrieve(taskId); - TaskExecutionLog execLog = new TaskExecutionLog(getProcessInfo().getHost(), - workerNumber); - execLog.setStartProcessingTime(new Date(startProcessingTimeMillis)); - execLog.setInitialProcessingStep(pipelineTask.getProcessingStep()); - pipelineTask.getExecLog().add(execLog); - return pipelineTaskCrud().merge(pipelineTask); - }); - } - - /** - * Creates instances of {@link RemoteJob} in the database using information obtained from the - * output of the remote cluster's "qstat" command. The instances are initialized to estimated - * cost of zero and unfinished status. - */ - public void createRemoteJobsFromQstat(long pipelineTaskId) { - - performTransaction(() -> { - PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); - PipelineTask databaseTask = pipelineTaskCrud.retrieve(pipelineTaskId); - QueueCommandManager queueCommandManager = queueCommandManager(); - QstatMonitor qstatMonitor = new QstatMonitor(queueCommandManager); - qstatMonitor.addToMonitoring(databaseTask); - qstatMonitor.update(); - Set allIncompleteJobIds = qstatMonitor.allIncompleteJobIds(databaseTask); - log.info("Job IDs for task " + databaseTask.getId() + " from qstat : " - + allIncompleteJobIds.toString()); - Set remoteJobs = databaseTask.getRemoteJobs(); - for (long jobId : allIncompleteJobIds) { - remoteJobs.add(new RemoteJob(jobId)); - } - }); - } - - QueueCommandManager queueCommandManager() { - return QueueCommandManager.newInstance(); - } - - public PipelineTask updateJobs(PipelineTask pipelineTask) { - return updateJobs(pipelineTask, false); - } - - /** - * Updates the state of all {@link RemoteJob}s associated with a {@link PipelineTask}. Any jobs - * that have completed since the last update will be marked as finished and get their final cost - * estimates calculated; any that are still running will get an up-to-the-minute cost estimate - * calculated. If the boolean argument is true, all jobs associated with the pipeline task will - * be moved to the finished state. This is useful when the pipeline task algorithm has - * completed, and we want to ensure that all the jobs that were used for the task are correctly - * recorded as complete. - * - * @return a new object with an updated remoteJobs property - */ - public PipelineTask updateJobs(PipelineTask pipelineTask, boolean markJobsCompleted) { - - QueueCommandManager queueCommandManager = queueCommandManager(); - Set remoteJobs = remoteJobs(pipelineTask); - for (RemoteJob job : remoteJobs) { - if (job.isFinished()) { - continue; - } - RemoteJob.RemoteJobQstatInfo jobInfo = queueCommandManager - .remoteJobQstatInfo(job.getJobId()); - job.setCostEstimate(jobInfo.costEstimate()); - - // Is the job finished? - if (markJobsCompleted ? true : queueCommandManager.exitStatus(job.getJobId()) != null) { - Log.info("Job " + job.getJobId() + " marked as finished"); - job.setFinished(true); - log.info("Job " + job.getJobId() + " cost estimate: " + job.getCostEstimate()); - } else { - log.info("Incomplete job " + job.getJobId() + " running cost estimate: " - + job.getCostEstimate()); - } - } - - return performTransaction(() -> { - PipelineTask databaseTask = pipelineTask(pipelineTask.getId()); - databaseTask.setRemoteJobs(remoteJobs); - return pipelineTaskCrud().merge(databaseTask); - }); - } - - Set remoteJobs(PipelineTask pipelineTask) { return performTransaction( - () -> pipelineTaskCrud().retrieveRemoteJobs(pipelineTask.getId())); + () -> pipelineTaskCrud().retrieveTasksForInstance(pipelineInstance)); } public String pipelineDefinitionName(PipelineTask pipelineTask) { @@ -360,162 +162,11 @@ public String pipelineDefinitionName(PipelineTask pipelineTask) { .getName()); } - public List taskIdsForPipelineDefinitionNode(PipelineTask pipelineTask) { - return performTransaction(() -> pipelineTaskCrud().retrieveIdsForPipelineDefinitionNode( + public List tasksForPipelineDefinitionNode(PipelineTask pipelineTask) { + return performTransaction(() -> pipelineTaskCrud().retrieveTasksForPipelineDefinitionNode( pipelineTaskCrud().retrievePipelineDefinitionNode(pipelineTask))); } - /** - * Updates the subtask counts in the database. - */ - public void updateSubtaskCounts(long taskId, int totalSubtaskCount, int completedSubtaskCount, - int failedSubtaskCount) { - performTransaction(() -> { - PipelineTaskCrud crud = new PipelineTaskCrud(); - PipelineTask task = crud.retrieve(taskId); - task.setCompletedSubtaskCount(completedSubtaskCount); - task.setFailedSubtaskCount(failedSubtaskCount); - task.setTotalSubtaskCount(totalSubtaskCount); - crud.merge(task); - }); - } - - /** - * Updates the processing step of a {@link PipelineTask} and simultaneously updates the state of - * that task's {@link PipelineInstance} if need be. This is a safer method to use than the - * "bare" {@link PipelineTask#setProcessingStep(ProcessingStep)} because it also automatically - * updates the pipeline instance, and starts or stops execution clocks as needed. - *

- * This method first retrieves the pipeline task from the database; it then performs the updates - * and merges the changes back to the database. Thus, this is a better method to use in cases - * where a local copy of the pipeline task has changes that should not be merged to the - * database. - * - * @return the updated and merged pipeline task, which must be used in subsequent processing - */ - public PipelineTask updateProcessingStep(long taskId, ProcessingStep processingStep) { - return updateProcessingStep(pipelineTaskCrud().retrieve(taskId), processingStep); - } - - /** - * Updates the processing step of a {@link PipelineTask} and simultaneously updates the state of - * that task's {@link PipelineInstance} if need be. This is a safer method to use than the - * "bare" {@link PipelineTask#setProcessingStep(ProcessingStep)} because it also automatically - * updates the pipeline instance, and starts or stops execution clocks as needed. - *

- * This is the method to use if you have a detached pipeline task that contains changes which - * should be persisted. - * - * @return the updated and merged pipeline task, which must be used in subsequent processing - */ - public PipelineTask updateProcessingStep(PipelineTask task, ProcessingStep processingStep) { - return performTransaction(() -> { - task.setProcessingStep(processingStep); - if (processingStep == ProcessingStep.INITIALIZING - || processingStep == ProcessingStep.COMPLETE) { - task.stopExecutionClock(); - } else { - task.startExecutionClock(); - } - PipelineTask mergedTask = pipelineTaskCrud().merge(task); - updateInstanceState(mergedTask); - - return mergedTask; - }); - } - - /** - * Sets the {@link PipelineTask}'s error flag and simultaneously updates the state of that - * task's {@link PipelineInstance} if need be. This is a safer method to use than the "bare" - * {@link PipelineTask#setProcessingStep(gov.nasa.ziggy.pipeline.definition.ProcessingStep)} - * because it also automatically updates the pipeline instance, and starts or stops execution - * clocks as needed. - *

- * This method first retrieves the pipeline task from the database; it then performs the updates - * and merges the changes back to the database. Thus, this is a better method to use in cases - * where a local copy of the pipeline task has changes that should not be merged to the - * database. - * - * @return the updated and merged pipeline task, which must be used in subsequent processing - */ - public PipelineTask taskErrored(long taskId) { - return taskErrored(pipelineTaskCrud().retrieve(taskId)); - } - - /** - * Sets the {@link PipelineTask}'s error flag and simultaneously updates the state of that - * task's {@link PipelineInstance} if need be. This is a safer method to use than the "bare" - * {@link PipelineTask#setError(boolean)} because it also automatically updates the pipeline - * instance, and stops the execution clock. - *

- * This is the method to use if you have a detached pipeline task that contains changes which - * should be persisted. - * - * @return the updated and merged pipeline task, which must be used in subsequent processing - */ - public PipelineTask taskErrored(PipelineTask task) { - return performTransaction(() -> { - task.setError(true); - task.stopExecutionClock(); - PipelineTask mergedTask = pipelineTaskCrud().merge(task); - updateInstanceState(mergedTask); - - return mergedTask; - }); - } - - private PipelineInstance updateInstanceState(PipelineTask mergedTask) { - - PipelineInstance pipelineInstance = pipelineTaskCrud().retrievePipelineInstance(mergedTask); - - PipelineInstanceNode pipelineInstanceNode = pipelineTaskCrud() - .retrievePipelineInstanceNode(mergedTask); - TaskCounts instanceNodeCounts = pipelineInstanceNodeOperations() - .taskCounts(pipelineInstanceNode); - - PipelineInstance.State state = PipelineInstance.State.INITIALIZED; - if (new PipelineInstanceOperations().allInstanceNodesExecutionComplete(pipelineInstance)) { - - // If all the instance nodes are done, we can set the instance state to - // either completed or stalled. - state = instanceNodeCounts.isPipelineTasksComplete() ? PipelineInstance.State.COMPLETED - : PipelineInstance.State.ERRORS_STALLED; - } else if (instanceNodeCounts.isPipelineTasksExecutionComplete()) { - - // If the current node is done, then the state is either stalled or processing. - state = instanceNodeCounts.isPipelineTasksComplete() ? PipelineInstance.State.PROCESSING - : PipelineInstance.State.ERRORS_STALLED; - } else { - - // If the current instance node is still grinding away, then the state is either - // errors running or processing - state = instanceNodeCounts.getTotalCounts().getFailedTaskCount() == 0 - ? PipelineInstance.State.PROCESSING - : PipelineInstance.State.ERRORS_RUNNING; - } - - state.setExecutionClockState(pipelineInstance); - pipelineInstance.setState(state); - - return pipelineInstanceCrud().merge(pipelineInstance); - } - - public PipelineTask clearError(long taskId) { - return performTransaction(() -> { - PipelineTask task = pipelineTaskCrud().retrieve(taskId); - task.clearError(); - return pipelineTaskCrud().merge(task); - }); - } - - public void incrementPipelineTaskLogIndex(long taskId) { - performTransaction(() -> { - PipelineTask task = pipelineTaskCrud().retrieve(taskId); - task.incrementTaskLogIndex(); - pipelineTaskCrud.merge(task); - }); - } - public PipelineDefinitionNode pipelineDefinitionNode(PipelineTask pipelineTask) { return performTransaction( () -> pipelineTaskCrud().retrievePipelineDefinitionNode(pipelineTask)); @@ -549,6 +200,11 @@ public PipelineInstance pipelineInstance(PipelineTask pipelineTask) { return performTransaction(() -> pipelineTaskCrud().retrievePipelineInstance(pipelineTask)); } + public PipelineInstance pipelineInstance(long pipelineTaskId) { + return performTransaction( + () -> pipelineTaskCrud().retrievePipelineInstance(pipelineTaskId)); + } + public PipelineModuleDefinition pipelineModuleDefinition(PipelineTask pipelineTask) { return performTransaction( () -> pipelineTaskCrud().retrievePipelineModuleDefinition(pipelineTask)); @@ -609,6 +265,14 @@ PipelineTaskCrud pipelineTaskCrud() { return pipelineTaskCrud; } + PipelineTaskDataCrud pipelineTaskDataCrud() { + return pipelineTaskDataCrud; + } + + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud() { return pipelineDefinitionNodeCrud; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactory.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactory.java index 54a3732..a11ac59 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactory.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactory.java @@ -38,6 +38,7 @@ public class RuntimeObjectFactory extends DatabaseOperations { private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); private PipelineDefinitionOperations pipelineDefinitionOperations = new PipelineDefinitionOperations(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); /** * Creates {@link PipelineInstanceNode} instances for the initial @@ -149,10 +150,10 @@ public List newPipelineTasks(PipelineInstanceNode instanceNode, PipelineInstance instance, List unitsOfWork) { List pipelineTasks = new ArrayList<>(); for (UnitOfWork unitOfWork : unitsOfWork) { - PipelineTask pipelineTask = new PipelineTask(instance, instanceNode); - pipelineTask.setProcessingStep(ProcessingStep.WAITING_TO_RUN); - pipelineTask.setUowTaskParameters(unitOfWork.getParameters()); + PipelineTask pipelineTask = new PipelineTask(instance, instanceNode, unitOfWork); pipelineTask = pipelineTaskOperations().merge(pipelineTask); + pipelineTaskDataOperations().createPipelineTaskData(pipelineTask, + ProcessingStep.WAITING_TO_RUN); pipelineTasks.add(pipelineTask); } pipelineInstanceNodeOperations().addPipelineTasks(instanceNode, pipelineTasks); @@ -186,4 +187,8 @@ PipelineDefinitionOperations pipelineDefinitionOperations() { PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/UniqueNameVersionPipelineComponentCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/UniqueNameVersionPipelineComponentCrud.java index 2307fd1..94f034e 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/database/UniqueNameVersionPipelineComponentCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/database/UniqueNameVersionPipelineComponentCrud.java @@ -35,10 +35,6 @@ public U retrieveLatestVersionForName(String name) { query.column(UniqueNameVersionPipelineComponent_.NAME).in(name); query.column(UniqueNameVersionPipelineComponent_.VERSION).in(versionQuery); U result = uniqueResult(query); - if (result == null) { - return null; - } - return result; } @@ -46,8 +42,7 @@ public U retrieve(String name, int version) { ZiggyQuery query = createZiggyQuery(componentClass()); query.column(UniqueNameVersionPipelineComponent_.NAME).in(name); query.column(UniqueNameVersionPipelineComponent_.VERSION).in(version); - U result = uniqueResult(query); - return result; + return uniqueResult(query); } /** @@ -56,8 +51,7 @@ public U retrieve(String name, int version) { public List retrieveAllVersionsForName(String name) { ZiggyQuery query = createZiggyQuery(componentClass()); query.column(UniqueNameVersionPipelineComponent_.NAME).in(name); - List results = list(query); - return results; + return list(query); } /** @@ -96,52 +90,6 @@ public boolean databaseContainsName(U component) { return uniqueResult(query) > 0; } - /** - * Renames an unlocked instance. Locked instances are not allowed to be renamed. - *

- * The new name must not be a name that is already in use in the database. Because the object is - * saved to a new name, its version is set to zero and its lock status is set to unlocked. - * - * @return the merged instance, which should be used in lieu of {@code pipelineComponent} - */ - public final U rename(U pipelineComponent, String newName) { - String oldName = pipelineComponent.getName(); - - if (pipelineComponent.getVersion() > 0 || pipelineComponent.isLocked()) { - throw new PipelineException("Unable to change name of " - + componentNameForExceptionMessages() + " from " + oldName + " to " + newName - + " as " + oldName + " is locked or its version is greater than zero"); - } - - // If the name is changed, the new name isn't allowed to conflict with - // any existing pipeline definition names. - if (!newName.equals(oldName) && retrieveLatestVersionForName(newName) != null) { - throw new PipelineException("Unable to change name of " - + componentNameForExceptionMessages() + " from " + oldName + " to " + newName - + " due to conflict with existing definition in database"); - } - - pipelineComponent.setName(newName); - return super.merge(pipelineComponent); - } - - /** - * Removes an unlocked instance. Locked instances are not allowed to be deleted. - */ - @Override - public final void remove(Object o) { - if (o instanceof UniqueNameVersionPipelineComponent) { - UniqueNameVersionPipelineComponent pipelineComponent = (UniqueNameVersionPipelineComponent) o; - if (pipelineComponent.getVersion() > 0 || pipelineComponent.isLocked()) { - throw new PipelineException("Unable to remove " - + componentNameForExceptionMessages() + " as " + pipelineComponent.getName() - + " is locked or its version is greater than zero"); - } - } - - super.remove(o); - } - /** * Persists or merges an object into the database, as appropriate, with persist semantics. That * means that updates to the persisted object will be persisted as well. diff --git a/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibrary.java b/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibrary.java index f8eb1d2..d097ea3 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibrary.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibrary.java @@ -10,10 +10,11 @@ import org.apache.commons.configuration2.ImmutableConfiguration; import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.BuildInfo; +import gov.nasa.ziggy.util.BuildInfo.BuildType; import jakarta.xml.bind.annotation.XmlAccessType; import jakarta.xml.bind.annotation.XmlAccessorType; import jakarta.xml.bind.annotation.XmlAttribute; @@ -53,7 +54,7 @@ public class ParameterLibrary implements HasXmlSchemaFilename { public ParameterLibrary() { ImmutableConfiguration config = ZiggyConfiguration.getInstance(); - release = config.getString(PropertyName.ZIGGY_VERSION.property()); + release = new BuildInfo(BuildType.PIPELINE).getSoftwareVersion(); databaseUrl = config.getString(HIBERNATE_URL.property(), ""); databaseUser = config.getString(HIBERNATE_USERNAME.property(), ""); overrideOnly = false; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibraryImportExportCli.java b/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibraryImportExportCli.java index de660f1..a16a389 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibraryImportExportCli.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/xml/ParameterLibraryImportExportCli.java @@ -80,7 +80,7 @@ public void go() { List results = null; for (String filename : filenames) { - log.debug("Importing: " + filename); + log.debug("Importing {}", filename); if (importLib) { results = paramOps.importParameterLibrary(filename, null, paramIoMode); } else { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java index 8e24117..1094b87 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java @@ -87,10 +87,10 @@ public static void main(String[] args) { destinationDirectory = Paths.get(xmlDir, "xml").toString(); new File(destinationDirectory).mkdirs(); for (Class clazz : XmlSchemaExporter.schemaClasses()) { - log.info("Generating XML schema for class " + clazz.getName() + "..."); + log.info("Generating XML schema for class {}", clazz.getName()); JAXBContext context = JAXBContext.newInstance(clazz); context.generateSchema(new PipelineDefinitionSchemaResolver(clazz)); - log.info("Generating XML schema for class " + clazz.getName() + "...done"); + log.info("Generating XML schema for class {}...done", clazz.getName()); } } catch (IOException | JAXBException e) { log.error("Unable to generate schema", e); diff --git a/src/main/java/gov/nasa/ziggy/services/alert/Alert.java b/src/main/java/gov/nasa/ziggy/services/alert/Alert.java index 65b9406..f2e651f 100644 --- a/src/main/java/gov/nasa/ziggy/services/alert/Alert.java +++ b/src/main/java/gov/nasa/ziggy/services/alert/Alert.java @@ -5,11 +5,14 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.slf4j.event.Level; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.messages.AlertMessage; import jakarta.persistence.Column; import jakarta.persistence.Embeddable; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.ManyToOne; /** * Contains alert data. Shared by {@link Alert} (used to store alert data in the database) and @@ -19,43 +22,43 @@ */ @Embeddable public class Alert implements Serializable { + + public enum Severity { + INFRASTRUCTURE, ERROR, WARNING; + } + private static final Logger log = LoggerFactory.getLogger(Alert.class); - private static final long serialVersionUID = 20230511L; + private static final long serialVersionUID = 20240925L; private static final int MAX_MESSAGE_LENGTH = 4000; private Date timestamp; - private String sourceComponent = null; - private long sourceTaskId = -1; - private String processName = null; - private String processHost = null; + private String sourceComponent; + + /** The source task is null when this object is deserialized. */ + @ManyToOne + private PipelineTask sourceTask; + + private String processName; + private String processHost; private long processId = -1; - private String severity = Level.ERROR.toString(); - @Column(nullable = true, length = MAX_MESSAGE_LENGTH) - private String message = null; - public Alert() { - } + @Enumerated(EnumType.STRING) + private Severity severity = Severity.ERROR; - public Alert(Date timestamp, String sourceComponent, long sourceTaskId, String processName, - String processHost, long processId, String message) { - this.timestamp = timestamp; - this.sourceComponent = sourceComponent; - this.sourceTaskId = sourceTaskId; - this.processName = processName; - this.processHost = processHost; - this.processId = processId; - this.message = message; + @Column(nullable = true, length = MAX_MESSAGE_LENGTH) + private String message; - validateMessageLength(); + // For Hibernate. + Alert() { } - public Alert(Date timestamp, String sourceComponent, long sourceTaskId, String processName, - String processHost, long processId, String severity, String message) { + public Alert(Date timestamp, String sourceComponent, PipelineTask sourceTask, + String processName, String processHost, long processId, Severity severity, String message) { this.timestamp = timestamp; this.sourceComponent = sourceComponent; - this.sourceTaskId = sourceTaskId; + this.sourceTask = sourceTask; this.processName = processName; this.processHost = processHost; this.processId = processId; @@ -71,8 +74,8 @@ private void validateMessageLength() { log.warn("Alert message is NULL"); } else if (message.length() > MAX_MESSAGE_LENGTH) { message = message.substring(0, MAX_MESSAGE_LENGTH - 4) + "..."; - log.warn("Alert message length (" + message.length() + ") is too long, max = " - + MAX_MESSAGE_LENGTH + ", truncated"); + log.warn("Alert message length ({}) is too long, max is {}, truncated", + message.length(), MAX_MESSAGE_LENGTH); } } @@ -100,12 +103,12 @@ public void setSourceComponent(String sourceComponent) { this.sourceComponent = sourceComponent; } - public long getSourceTaskId() { - return sourceTaskId; + public PipelineTask getSourceTask() { + return sourceTask; } - public void setSourceTaskId(long sourceTaskId) { - this.sourceTaskId = sourceTaskId; + public void setSourceTask(PipelineTask sourceTask) { + this.sourceTask = sourceTask; } public Date getTimestamp() { @@ -132,15 +135,11 @@ public void setMessage(String message) { this.message = message; } - public String getSeverity() { + public Severity getSeverity() { return severity; } - public void setSeverity(Level severity) { - this.severity = severity.toString(); - } - - public void setSeverity(String severity) { + public void setSeverity(Severity severity) { this.severity = severity; } } diff --git a/src/main/java/gov/nasa/ziggy/services/alert/AlertLogCrud.java b/src/main/java/gov/nasa/ziggy/services/alert/AlertLogCrud.java index 9d7658f..2c85d64 100644 --- a/src/main/java/gov/nasa/ziggy/services/alert/AlertLogCrud.java +++ b/src/main/java/gov/nasa/ziggy/services/alert/AlertLogCrud.java @@ -3,19 +3,19 @@ import static com.google.common.base.Preconditions.checkNotNull; import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; import java.util.Date; import java.util.List; -import java.util.stream.Collectors; import org.hibernate.HibernateException; -import gov.nasa.ziggy.collections.ListChunkIterator; +import com.google.common.collect.Lists; + import gov.nasa.ziggy.crud.AbstractCrud; import gov.nasa.ziggy.crud.ZiggyQuery; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskCrud; +import gov.nasa.ziggy.services.alert.Alert.Severity; /** * This class provides CRUD methods for the AlertService @@ -49,12 +49,12 @@ public List retrieveComponents() { * @throws NullPointerException if any of the arguments were {@code null} * @throws HibernateException if there were problems accessing the database */ - public List retrieveSeverities() { - ZiggyQuery query = createZiggyQuery(AlertLog.class, String.class); + public List retrieveSeverities() { + ZiggyQuery query = createZiggyQuery(AlertLog.class, Severity.class); query.select(query.get(AlertLog_.alertData).get(Alert_.severity)); query.getCriteriaQuery() - .orderBy( - query.getBuilder().asc(query.getRoot().get("alertData"). get("severity"))); + .orderBy(query.getBuilder() + .asc(query.getRoot().get(AlertLog_.alertData).get(Alert_.severity))); query.distinct(true); return list(query); } @@ -69,7 +69,7 @@ public List retrieveSeverities() { * @throws HibernateException if there were problems accessing the database */ public List retrieve(Date startDate, Date endDate) { - return retrieve(startDate, endDate, new String[0], new String[0]); + return retrieve(startDate, endDate, List.of(), List.of()); } /** @@ -84,8 +84,8 @@ public List retrieve(Date startDate, Date endDate) { * @throws NullPointerException if any of the arguments were {@code null} * @throws HibernateException if there were problems accessing the database */ - public List retrieve(Date startDate, Date endDate, String[] components, - String[] severities) { + public List retrieve(Date startDate, Date endDate, List components, + List severities) { checkNotNull(components, "components"); checkNotNull(severities, "severities"); checkNotNull(startDate, "startDate"); @@ -94,13 +94,12 @@ public List retrieve(Date startDate, Date endDate, String[] components ZiggyQuery query = createZiggyQuery(AlertLog.class); query.where(query.getBuilder() .between(query.get(AlertLog_.alertData).get(Alert_.timestamp), startDate, endDate)); - if (components.length > 0) { - query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.sourceComponent), - Arrays.asList(components))); + if (!components.isEmpty()) { + query.where( + query.in(query.get(AlertLog_.alertData).get(Alert_.sourceComponent), components)); } - if (severities.length > 0) { - query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.severity), - Arrays.asList(severities))); + if (!severities.isEmpty()) { + query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.severity), severities)); } query.getCriteriaQuery() @@ -114,39 +113,30 @@ public List retrieve(Date startDate, Date endDate, String[] components /** * Retrieve all alerts for the specified pipeline instance. - * - * @param pipelineInstanceId - * @return */ - public List retrieveForPipelineInstance(long pipelineInstanceId) { + public List retrieveForPipelineInstance(PipelineInstance pipelineInstance) { // I don't know how to do this as one query, so I'm doing it as two. List tasksInInstance = new PipelineTaskCrud() - .retrieveTasksForInstance(pipelineInstanceId); - List taskIds = tasksInInstance.stream() - .map(PipelineTask::getId) - .collect(Collectors.toList()); + .retrieveTasksForInstance(pipelineInstance); ZiggyQuery query = createZiggyQuery(AlertLog.class); - query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.sourceTaskId), taskIds)); + query.where( + query.in(query.get(AlertLog_.alertData).get(Alert_.sourceTask), tasksInInstance)); return list(query); } - /** - * Retrieve all alerts for the specified list of pipeline task ids. - */ - public List retrieveByPipelineTaskIds(Collection taskIds) { - List rv = new ArrayList<>(); - ListChunkIterator idIt = new ListChunkIterator<>(taskIds.iterator(), 50); - for (List idChunk : idIt) { - rv.addAll(retrieveChunk(idChunk)); + public List retrieveByPipelineTasks(List tasks) { + List alertLogs = new ArrayList<>(); + for (List idChunk : Lists.partition(tasks, 50)) { + alertLogs.addAll(retrieveChunkByPipelineTasks(idChunk)); } - return rv; + return alertLogs; } - private List retrieveChunk(List taskIds) { + private List retrieveChunkByPipelineTasks(List tasks) { ZiggyQuery query = createZiggyQuery(AlertLog.class); - query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.sourceTaskId), taskIds)); + query.where(query.in(query.get(AlertLog_.alertData).get(Alert_.sourceTask), tasks)); return list(query); } diff --git a/src/main/java/gov/nasa/ziggy/services/alert/AlertLogOperations.java b/src/main/java/gov/nasa/ziggy/services/alert/AlertLogOperations.java index 5ed0325..b93ea76 100644 --- a/src/main/java/gov/nasa/ziggy/services/alert/AlertLogOperations.java +++ b/src/main/java/gov/nasa/ziggy/services/alert/AlertLogOperations.java @@ -2,23 +2,35 @@ import java.util.List; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceCrud; import gov.nasa.ziggy.services.database.DatabaseOperations; /** Operations class for alerts. */ public class AlertLogOperations extends DatabaseOperations { private AlertLogCrud alertLogCrud = new AlertLogCrud(); + private PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); public void persist(AlertLog alert) { performTransaction(() -> alertLogCrud().persist(alert)); } - public List alertLogs(long pipelineInstanceId) { + public List alertLogs(PipelineInstance pipelineInstance) { return performTransaction( - () -> alertLogCrud().retrieveForPipelineInstance(pipelineInstanceId)); + () -> alertLogCrud().retrieveForPipelineInstance(pipelineInstance)); + } + + public List alertLogs(long pipelineInstanceId) { + return performTransaction(() -> alertLogCrud() + .retrieveForPipelineInstance(pipelineInstanceCrud().retrieve(pipelineInstanceId))); } AlertLogCrud alertLogCrud() { return alertLogCrud; } + + PipelineInstanceCrud pipelineInstanceCrud() { + return pipelineInstanceCrud; + } } diff --git a/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java b/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java index 06f7f24..76f08f7 100644 --- a/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java +++ b/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java @@ -9,6 +9,8 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.messages.AlertMessage; @@ -24,17 +26,8 @@ * @author Todd Klaus */ public class AlertService { - public enum Severity { - INFRASTRUCTURE, ERROR, WARNING; - - @Override - public String toString() { - return name(); - } - } - private static final Logger log = LoggerFactory.getLogger(AlertService.class); - public static final int DEFAULT_TASK_ID = -1; + public static final PipelineTask DEFAULT_TASK = null; public static final boolean BROADCAST_ALERTS_ENABLED_DEFAULT = false; public boolean broadcastEnabled = false; @@ -59,36 +52,27 @@ public AlertService() { BROADCAST_ALERTS_ENABLED_DEFAULT); } - public void generateAlert(String sourceComponent, String message) { - generateAlert(sourceComponent, DEFAULT_TASK_ID, message); + public void generateAlert(String sourceComponent, PipelineTask sourceTask, String message) { + generateAlert(sourceComponent, sourceTask, Severity.ERROR, message); } - public void generateAlert(String sourceComponent, AlertService.Severity severity, - String message) { - generateAlert(sourceComponent, DEFAULT_TASK_ID, severity, message); - } - - /** - * @see gov.nasa.ziggy.services.alert.AlertService#generateAlert(java.lang.String, long, - * java.lang.String) - */ - public void generateAlert(String sourceComponent, long sourceTaskId, String message) { - generateAlert(sourceComponent, sourceTaskId, AlertService.Severity.ERROR, message); + public void generateAlert(String sourceComponent, Severity severity, String message) { + generateAlert(sourceComponent, DEFAULT_TASK, severity, message); } - public void generateAndBroadcastAlert(String sourceComponent, long sourceTaskId, - AlertService.Severity severity, String message) { + public void generateAndBroadcastAlert(String sourceComponent, PipelineTask sourceTask, + Severity severity, String message) { boolean storedBroadcastFlag = broadcastEnabled; broadcastEnabled = true; - generateAlert(sourceComponent, sourceTaskId, severity, message); + generateAlert(sourceComponent, sourceTask, severity, message); broadcastEnabled = storedBroadcastFlag; } @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - public void generateAlert(String sourceComponent, long sourceTaskId, - AlertService.Severity severity, String message) { - log.debug("ALERT:[" + sourceComponent + "]: " + message); + public void generateAlert(String sourceComponent, PipelineTask sourceTask, Severity severity, + String message) { + log.debug("ALERT:[{}]: {}", sourceComponent, message); Date timestamp = new Date(); String processName = null; @@ -112,17 +96,16 @@ public void generateAlert(String sourceComponent, long sourceTaskId, } } - Alert alertData = new Alert(timestamp, sourceComponent, sourceTaskId, processName, - processHost, processId, severity.toString(), message); + Alert alertData = new Alert(timestamp, sourceComponent, sourceTask, processName, + processHost, processId, severity, message); - // store alert in db try { alertLogOperations().persist(new AlertLog(alertData)); } catch (PipelineException e) { - log.error("Failed to store Alert in database", e); + log.error("Failed to store alert in database", e); } - if (broadcastEnabled || severity == AlertService.Severity.INFRASTRUCTURE) { + if (broadcastEnabled || severity == Severity.INFRASTRUCTURE) { ZiggyMessenger.publish(new AlertMessage(alertData)); } } diff --git a/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java b/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java index 47c665e..8df3a28 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java +++ b/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java @@ -80,28 +80,28 @@ public void merge() { // Read base file File baseFile = new File(basePath); if (!baseFile.exists()) { - throw new PipelineException("baseFile: " + baseFile + " does not exist"); + throw new PipelineException("baseFile " + baseFile + " does not exist"); } - log.info("reading base file: " + baseFile); + log.info("Reading base file {}", baseFile); PropertiesConfiguration baseConfig = new Configurations().properties(baseFile); // Read the override file File overrideFile = new File(overridePath); if (!overrideFile.exists()) { - throw new PipelineException("overrideFile: " + overrideFile + " does not exist"); + throw new PipelineException("overrideFile " + overrideFile + " does not exist"); } - log.info("reading override file: " + overrideFile); + log.info("Reading override file {}", overrideFile); PropertiesConfiguration overrideConfig = new Configurations().properties(overrideFile); // Copy the override properties to the base config ConfigurationUtils.copy(overrideConfig, baseConfig); // Write out the merged config .properties file - log.info("writing merged file: " + mergedFile); - BufferedWriter outputWriter = new BufferedWriter( - new OutputStreamWriter(new FileOutputStream(mergedFile), ZiggyFileUtils.ZIGGY_CHARSET)); + log.info("Writing merged file {}", mergedFile); + BufferedWriter outputWriter = new BufferedWriter(new OutputStreamWriter( + new FileOutputStream(mergedFile), ZiggyFileUtils.ZIGGY_CHARSET)); baseConfig.write(outputWriter); } catch (IOException e) { throw new UncheckedIOException("Unable to write to file " + mergedFile.toString(), e); diff --git a/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java b/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java index b52bf2e..da86209 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java +++ b/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java @@ -18,7 +18,6 @@ public class DirectoryProperties { private static final String TASK_LOG_FILES_RELATIVE_PATH = "ziggy"; private static final String CLI_LOG_FILES_RELATIVE_PATH = "cli"; private static final String PBS_LOG_FILES_RELATIVE_PATH = "pbs"; - private static final String STATE_FILES_RELATIVE_PATH = "state"; private static final String ALGORITHM_LOG_FILES_RELATIVE_PATH = "algorithms"; private static final String DATABASE_LOG_FILES_RELATIVE_PATH = "db"; private static final String SUPERVISOR_LOG_FILES_RELATIVE_PATH = "supervisor"; @@ -73,11 +72,6 @@ public static Path pbsLogDir() { .resolve(PBS_LOG_FILES_RELATIVE_PATH); } - public static Path stateFilesDir() { - return pipelineResultsDir().resolve(LOG_FILES_RELATIVE_PATH) - .resolve(STATE_FILES_RELATIVE_PATH); - } - public static Path algorithmLogsDir() { return pipelineResultsDir().resolve(LOG_FILES_RELATIVE_PATH) .resolve(ALGORITHM_LOG_FILES_RELATIVE_PATH); diff --git a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePair.java b/src/main/java/gov/nasa/ziggy/services/config/KeyValuePair.java deleted file mode 100644 index 684357f..0000000 --- a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePair.java +++ /dev/null @@ -1,70 +0,0 @@ -package gov.nasa.ziggy.services.config; - -import java.util.Objects; - -import jakarta.persistence.Column; -import jakarta.persistence.Entity; -import jakarta.persistence.Id; -import jakarta.persistence.Table; -import jakarta.persistence.Version; - -@Entity -@Table(name = "ziggy_KeyValuePair") -public class KeyValuePair { - @Id - @Column(name = "keyName", length = 100) - private String key; - private String value; - - /** - * Used by Hibernate to implement optimistic locking. Should prevent 2 different console users - * from clobbering each others changes. - */ - @Version - private final int dirty = 0; - - public KeyValuePair() { - } - - public KeyValuePair(String key, String value) { - this.key = key; - this.value = value; - } - - public String getKey() { - return key; - } - - public void setKey(String key) { - this.key = key; - } - - public String getValue() { - return value; - } - - public void setValue(String value) { - this.value = value; - } - - public int getDirty() { - return dirty; - } - - @Override - public int hashCode() { - return Objects.hash(key, value); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if ((obj == null) || (getClass() != obj.getClass())) { - return false; - } - KeyValuePair other = (KeyValuePair) obj; - return Objects.equals(key, other.key) && Objects.equals(value, other.value); - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairCrud.java b/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairCrud.java deleted file mode 100644 index 589ff23..0000000 --- a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairCrud.java +++ /dev/null @@ -1,47 +0,0 @@ -package gov.nasa.ziggy.services.config; - -import java.util.List; - -import gov.nasa.ziggy.crud.AbstractCrud; -import gov.nasa.ziggy.crud.ZiggyQuery; - -/** - * Provides CRUD methods for the KeyValuePair entity - *

- * Note that an updateKeyValuePair() method is not needed! Changes to a persisted object will be - * updated in the database automatically when the transaction is committed, unless the object is - * detached. - * - * @author Todd Klaus - */ -public class KeyValuePairCrud extends AbstractCrud { - public List retrieveAll() { - return list(createZiggyQuery(KeyValuePair.class)); - } - - public String retrieveValue(String key) { - ZiggyQuery query = createZiggyQuery(KeyValuePair.class, String.class); - query.column(KeyValuePair_.key).in(key); - query.column(KeyValuePair_.value).select(); - return uniqueResult(query); - } - - public KeyValuePair retrieve(String key) { - ZiggyQuery query = createZiggyQuery(KeyValuePair.class); - query.column(KeyValuePair_.key).in(key); - return uniqueResult(query); - } - - public void create(KeyValuePair keyValuePair) { - super.persist(keyValuePair); - } - - public void delete(KeyValuePair keyValuePair) { - super.remove(keyValuePair); - } - - @Override - public Class componentClass() { - return KeyValuePair.class; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairOperations.java b/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairOperations.java deleted file mode 100644 index 8a47204..0000000 --- a/src/main/java/gov/nasa/ziggy/services/config/KeyValuePairOperations.java +++ /dev/null @@ -1,43 +0,0 @@ -package gov.nasa.ziggy.services.config; - -import java.util.List; - -import gov.nasa.ziggy.services.database.DatabaseOperations; - -/** - * Provides access to the {@link KeyValuePair} entity. - * - * @author Todd Klaus - * @author Bill Wohler - */ -public class KeyValuePairOperations extends DatabaseOperations { - private KeyValuePairCrud keyValuePairCrud = new KeyValuePairCrud(); - - public void persist(KeyValuePair keyValuePair) { - performTransaction(() -> keyValuePairCrud.persist(keyValuePair)); - } - - public KeyValuePair keyValuePair(String key) { - return performTransaction(() -> keyValuePairCrud.retrieve(key)); - } - - public List keyValuePairs() { - return performTransaction(() -> keyValuePairCrud.retrieveAll()); - } - - public String keyValuePairValue(String key) { - return performTransaction(() -> keyValuePairCrud.retrieve(key).getValue()); - } - - public KeyValuePair updateKeyValuePair(String key, String newValue) { - return performTransaction(() -> { - KeyValuePair keyValuePair = new KeyValuePairCrud().retrieve(key); - keyValuePair.setValue(newValue); - return new KeyValuePairCrud().merge(keyValuePair); - }); - } - - public void delete(KeyValuePair keyValuePair) { - performTransaction(() -> new KeyValuePairCrud().delete(keyValuePair)); - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java b/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java index 7904508..1134532 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java +++ b/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java @@ -278,9 +278,6 @@ public enum PropertyName { /** Allows the user to specify a working directory other than user.dir. For testing only. */ ZIGGY_TEST_WORKING_DIR("ziggy.test.working.dir"), - /** The current Ziggy version as determined by {@code git describe}. */ - ZIGGY_VERSION("ziggy.version"), - /** Allow persisting to continue although one or more subtasks failed. */ ALLOW_PARTIAL_TASKS("ziggy.worker.allowPartialTasks"), diff --git a/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java b/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java index d1e3bc9..77651bd 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java +++ b/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java @@ -6,13 +6,9 @@ import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_CONFIG_PATH; import java.io.File; -import java.io.IOException; -import java.io.UncheckedIOException; -import java.net.URL; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.util.Enumeration; import org.apache.commons.configuration2.CompositeConfiguration; import org.apache.commons.configuration2.Configuration; @@ -40,7 +36,6 @@ * @author Bill Wohler */ public class ZiggyConfiguration { - private static final String BUILD_CONFIGURATION = "ziggy-build.properties"; private static final Logger log = LoggerFactory.getLogger(ZiggyConfiguration.class); @@ -105,7 +100,6 @@ private static Configuration getConfiguration() { loadPipelineConfiguration(config); } loadZiggyConfiguration(config); - loadBuildConfiguration(config); return config; } @@ -169,28 +163,6 @@ private static void loadZiggyConfiguration(CompositeConfiguration config) { } } - /** - * Loads the build configuration from the {@value #BUILD_CONFIGURATION} file(s). - * - * @param config the non-{@code null} target composite configuration - */ - private static void loadBuildConfiguration(CompositeConfiguration config) { - try { - Enumeration buildUrl = ZiggyConfiguration.class.getClassLoader() - .getResources(BUILD_CONFIGURATION); - if (buildUrl.hasMoreElements()) { - while (buildUrl.hasMoreElements()) { - loadConfiguration(config, new File(buildUrl.nextElement().getPath())); - } - } else { - log.warn("Could not locate build information in {}", BUILD_CONFIGURATION); - } - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to load configuration from " + BUILD_CONFIGURATION, e); - } - } - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private static void loadConfiguration(CompositeConfiguration config, File propertiesFile) { try { diff --git a/src/main/java/gov/nasa/ziggy/services/database/AnnotatedPojoList.java b/src/main/java/gov/nasa/ziggy/services/database/AnnotatedPojoList.java index 2311a6b..62ee87c 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/AnnotatedPojoList.java +++ b/src/main/java/gov/nasa/ziggy/services/database/AnnotatedPojoList.java @@ -61,7 +61,7 @@ public Set> scanForClasses() { @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) public void processClass(ClassFile classFile) { if (isClassAnnotated(classFile)) { - // log.debug("Found annotated class: " + className); + // log.debug("Found annotated class {}", className); try { detectedClasses.add(Class.forName(classFile.getName())); } catch (ClassNotFoundException e) { @@ -79,7 +79,7 @@ private boolean isClassAnnotated(ClassFile classFile) { // if (cf.getName().endsWith(".package-info")) { // int idx = cf.getName().indexOf(".package-info"); // String pkgName = cf.getName().substring(0, idx); -// log.info("found package: " + pkgName); +// log.info("Found package ", pkgName); // packages.add(pkgName); // continue; // } @@ -89,18 +89,18 @@ private boolean isClassAnnotated(ClassFile classFile) { if (visible != null) { boolean isEntity = visible.getAnnotation(Entity.class.getName()) != null; if (isEntity) { - log.debug("found @Entity: " + classFile.getName()); + log.debug("Found @Entity {}", classFile.getName()); return true; } boolean isEmbeddable = visible.getAnnotation(Embeddable.class.getName()) != null; if (isEmbeddable) { - log.debug("found @Embeddable: " + classFile.getName()); + log.debug("Found @Embeddable {}", classFile.getName()); return true; } boolean isEmbeddableSuperclass = visible .getAnnotation(MappedSuperclass.class.getName()) != null; if (isEmbeddableSuperclass) { - log.debug("found @MappedSuperclass: " + classFile.getName()); + log.debug("Found @MappedSuperclass {}", classFile.getName()); return true; } } diff --git a/src/main/java/gov/nasa/ziggy/services/database/DatabaseController.java b/src/main/java/gov/nasa/ziggy/services/database/DatabaseController.java index 939ef5d..1a9b55f 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/DatabaseController.java +++ b/src/main/java/gov/nasa/ziggy/services/database/DatabaseController.java @@ -151,4 +151,9 @@ public String dbName() { * Checks whether the database specified by {@link #dbName()} exists. */ public abstract boolean dbExists(); + + /** + * Maximum expressions allowed in a query. + */ + public abstract int maxExpressions(); } diff --git a/src/main/java/gov/nasa/ziggy/services/database/DatabaseService.java b/src/main/java/gov/nasa/ziggy/services/database/DatabaseService.java index 126df99..c16693c 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/DatabaseService.java +++ b/src/main/java/gov/nasa/ziggy/services/database/DatabaseService.java @@ -29,7 +29,7 @@ public static synchronized DatabaseService getInstance() { if (!DatabaseService.usingLocalService.get()) { throw new PipelineException( - "A transaction was started but the " + "DatabaseService was not included."); + "A transaction was started but the DatabaseService was not included"); } if (instance != null) { diff --git a/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java b/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java index 29cf738..68a89ee 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java +++ b/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java @@ -54,6 +54,13 @@ public class HsqldbController extends DatabaseController { private static final Logger log = LoggerFactory.getLogger(HsqldbController.class); + /** + * This is the maximum number of dynamically-created expressions sent to the database. There + * doesn't seem to be a hard limit in HSQLDB, so pick the PostgreSQL value of 32,767. A setting + * of 95% of this values leaves plenty of room for other expressions in the query. + */ + private static final int MAX_EXPRESSIONS = (int) (32767 * 0.95); + private static final String SCHEMA_CREATE_FILE = "ddl.hsqldb-create.sql"; private static final String SCHEMA_DROP_FILE = "ddl.hsqldb-drop.sql"; private static final String DRIVER_CLASS_NAME = "org.hsqldb.jdbc.JDBCDriver"; @@ -171,7 +178,7 @@ private SqlRunner sqlRunner() { private void createInitTable() throws SQLException { try (Statement stmt = sqlRunner().connection().createStatement()) { String sqlString = String.format(CREATE_INIT_TABLE_SQL, INIT_TABLE_NAME); - log.debug("Executing SQL: {}", sqlString); + log.debug("Executing SQL {}", sqlString); stmt.executeUpdate(sqlString); stmt.getConnection().commit(); } @@ -180,7 +187,7 @@ private void createInitTable() throws SQLException { private void updateInitTable() throws SQLException { try (Statement stmt = sqlRunner().connection().createStatement()) { String sqlString = String.format(INSERT_INIT_TABLE_SQL, INIT_TABLE_NAME, new Date()); - log.debug("Executing SQL: {}", sqlString); + log.debug("Executing SQL {}", sqlString); stmt.executeUpdate(sqlString); stmt.getConnection().commit(); } @@ -228,7 +235,7 @@ private void dropDatabaseInternal() throws SQLException { if (tableCount > 0) { try { - log.debug("Executing script: {}", SCHEMA_DROP_FILE); + log.debug("Executing script {}", SCHEMA_DROP_FILE); executeScript(SCHEMA_DROP_FILE, true); } catch (Exception e) { throw new PipelineException("Drop script failed: " + e.getMessage(), e); @@ -246,7 +253,7 @@ private long tableCount() throws SQLException { String sqlString = String.format(TABLE_COUNT, INIT_TABLE_NAME); try (Statement stmt = sqlRunner().connection().createStatement(); ResultSet rs = stmt.executeQuery(sqlString)) { - log.debug("Executing SQL: {}", sqlString); + log.debug("Executing SQL {}", sqlString); if (rs.next()) { log.debug("{} -> {}", rs.getStatement().toString(), rs.getLong(1)); @@ -267,7 +274,7 @@ private List tableNames() throws SQLException { try (Statement stmt = sqlRunner().connection().createStatement(); ResultSet rs = stmt.executeQuery(sqlString)) { - log.debug("Executing SQL: {}", sqlString); + log.debug("Executing SQL {}", sqlString); while (rs.next()) { String tableName = rs.getString(1); tableNames.add(tableName); @@ -283,7 +290,7 @@ private boolean tableExists(String tableNameMixedCase) throws SQLException { String tableName = tableNameMixedCase.trim().toUpperCase(); String sqlString = String.format(TABLE_EXISTS, tableName); - log.debug("Executing SQL: {}", sqlString); + log.debug("Executing SQL {}", sqlString); try (Statement stmt = sqlRunner().connection().createStatement(); ResultSet rs = stmt.executeQuery(sqlString)) { if (rs.next()) { @@ -323,7 +330,7 @@ public int start() { } CommandLine supervisorStatusCommand = hsqldbCommand(WrapperCommand.START); - log.debug("Command line: " + supervisorStatusCommand.toString()); + log.debug("supervisorStatusCommand={}", supervisorStatusCommand.toString()); int exitStatus = ExternalProcess.simpleExternalProcess(supervisorStatusCommand) .exceptionOnFailure() .execute(); @@ -342,7 +349,7 @@ public int stop() { } CommandLine supervisorStopCommand = hsqldbCommand(WrapperCommand.STOP); - log.debug("Command line: " + supervisorStopCommand.toString()); + log.debug("supervisorStopCommand={}", supervisorStopCommand.toString()); return ExternalProcess.simpleExternalProcess(supervisorStopCommand).execute(true); } @@ -353,7 +360,7 @@ public int status() { } CommandLine supervisorStatusCommand = hsqldbCommand(WrapperCommand.STATUS); - log.debug("Command line: " + supervisorStatusCommand); + log.debug("supervisorStatusCommand={}", supervisorStatusCommand); return ExternalProcess.simpleExternalProcess(supervisorStatusCommand).execute(); } @@ -395,4 +402,9 @@ private void startServer() { server.setDatabasePath(0, dataDir().resolve(dbName()).toString()); server.start(); } + + @Override + public int maxExpressions() { + return MAX_EXPRESSIONS; + } } diff --git a/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java b/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java index 9e1ccce..3dd7cb3 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java +++ b/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java @@ -23,6 +23,7 @@ import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.os.OperatingSystemType; /** @@ -75,10 +76,10 @@ public static synchronized void initialize() { String log4jConfigFile = config .getString(PropertyName.LOG4J2_CONFIGURATION_FILE.property()); - log.info(PropertyName.LOG4J2_CONFIGURATION_FILE + " = " + log4jConfigFile); + log.info("{}={}", PropertyName.LOG4J2_CONFIGURATION_FILE, log4jConfigFile); if (log4jConfigFile != null) { - log.info("Log4j initialized with DOMConfigurator from: " + log4jConfigFile); + log.info("Log4j initialized with DOMConfigurator from {}", log4jConfigFile); // TODO Evaluate setting of log4j property // If ZiggyConfiguration.getInstance() is called before we get there, this // statement will have no effect. Consider rearchitecting so that this property @@ -94,11 +95,12 @@ public static synchronized void initialize() { } } - log.info("Software Version: {}", config.getString(PropertyName.ZIGGY_VERSION.property())); + log.info("Ziggy software version is {}", BuildInfo.ziggyVersion()); + log.info("Pipeline software version is {}", BuildInfo.pipelineVersion()); ZiggyConfiguration.logJvmProperties(); - long pid = OperatingSystemType.getInstance().getProcInfo().getPid(); - log.info("process ID: " + pid); + long pid = OperatingSystemType.newInstance().getProcInfo().getPid(); + log.info("Process ID is {}", pid); recordPid(pid); initialized = true; diff --git a/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java b/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java index ef258c8..dceb6db 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java +++ b/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java @@ -34,6 +34,13 @@ public class PostgresqlController extends DatabaseController { private static final Logger log = LoggerFactory.getLogger(DatabaseController.class); + /** + * This is the maximum number of dynamically-created expressions sent to the database. This + * limit is 32,767 in Postgresql. A setting of 95% of this value leaves plenty of room for other + * expressions in the query. + */ + private static final int MAX_EXPRESSIONS = (int) (32767 * 0.95); + private static final String PG_CTL = "pg_ctl"; private static final String INITDB = "initdb"; private static final String CREATEDB = "createdb"; @@ -98,7 +105,7 @@ public void createDatabase() { createCommand.addArgument("-p"); createCommand.addArgument(Integer.toString(port())); createCommand.addArgument(dbName()); - log.debug("Command line: " + createCommand.toString()); + log.debug("createCommand={}", createCommand.toString()); if (ExternalProcess.simpleExternalProcess(createCommand).execute(true) != 0) { throw new PipelineException("Unable to create database " + dbName()); } @@ -111,7 +118,7 @@ public void createDatabase() { schemaCommand.addArgument("-f"); schemaCommand.addArgument( DirectoryProperties.databaseSchemaDir().resolve(SCHEMA_CREATE_FILE).toString()); - log.debug("Command line: " + schemaCommand.toString()); + log.debug("schemaCommand={}", schemaCommand.toString()); ExternalProcess.simpleExternalProcess(schemaCommand).exceptionOnFailure().execute(true); File[] extraFiles = DirectoryProperties.databaseSchemaDir() @@ -122,7 +129,7 @@ public void createDatabase() { CommandLine extraSchemaCommand = psqlCommand(); extraSchemaCommand.addArgument("-f"); extraSchemaCommand.addArgument(extraFile.getAbsolutePath()); - log.debug("Command line: " + extraSchemaCommand.toString()); + log.debug("extraSchemaCommand={}", extraSchemaCommand.toString()); ExternalProcess.simpleExternalProcess(extraSchemaCommand) .exceptionOnFailure() .execute(true); @@ -143,7 +150,7 @@ private void initializeDatabase() { CommandLine commandLine = new CommandLine(commandStringWithPath(INITDB)); commandLine.addArgument("-D"); commandLine.addArgument(dataDir().toString()); - log.debug("Command line: " + commandLine.toString()); + log.debug("commandLine={}", commandLine.toString()); int retCode = ExternalProcess.simpleExternalProcess(commandLine).execute(true); if (retCode != 0) { throw new PipelineException("Database initialization failed"); @@ -163,7 +170,7 @@ private void updateConfigFile() { // Append to the postgresql.conf file an optional inclusion of the user's // config file and a setting of the lock file directory. - log.info("Creating configuration file in " + dataDir().toString()); + log.info("Creating configuration file in {}", dataDir().toString()); Path configPath = dataDir().resolve(CONFIG_FILE_NAME); try { @@ -180,9 +187,10 @@ private void updateConfigFile() { confFileContents .append("include '" + DirectoryProperties.databaseConfFile() + "'" + newline); } - Files.write(configPath, confFileContents.toString().getBytes(ZiggyFileUtils.ZIGGY_CHARSET), + Files.write(configPath, + confFileContents.toString().getBytes(ZiggyFileUtils.ZIGGY_CHARSET), StandardOpenOption.APPEND); - log.info("Creating configuration file in " + dataDir().toString() + "...done"); + log.info("Creating configuration file in {}...done", dataDir().toString()); } catch (IOException e) { throw new UncheckedIOException("Unable to write to file " + configPath.toString(), e); } @@ -194,7 +202,7 @@ public void dropDatabase() { schemaCommand.addArgument("-f"); schemaCommand.addArgument( DirectoryProperties.databaseSchemaDir().resolve(SCHEMA_DROP_FILE).toString()); - log.debug("Command line: " + schemaCommand.toString()); + log.debug("schemaCommand={}", schemaCommand.toString()); ExternalProcess.simpleExternalProcess(schemaCommand).exceptionOnFailure().execute(true); } @@ -216,7 +224,7 @@ public int start() { startCommand.addArgument("start"); startCommand.addArgument("-l"); startCommand.addArgument(logFile().toString()); - log.debug("Command line: " + startCommand.toString()); + log.debug("startCommand={}", startCommand.toString()); int status = ExternalProcess.simpleExternalProcess(startCommand) .exceptionOnFailure() .execute(); @@ -240,7 +248,7 @@ public int stop() { stopCommand.addArgument("-m"); stopCommand.addArgument("fast"); stopCommand.addArgument("stop"); - log.debug("Command line: " + stopCommand.toString()); + log.debug("stopCommand={}", stopCommand.toString()); return ExternalProcess.simpleExternalProcess(stopCommand).execute(true); } @@ -252,7 +260,7 @@ public int status() { CommandLine commandLine = pgctlCommandLine(true, true); commandLine.addArgument("status"); - log.debug("Command line: " + commandLine.toString()); + log.debug("commandLine={}", commandLine.toString()); return ExternalProcess.simpleExternalProcess(commandLine).execute(true); } @@ -263,7 +271,7 @@ public boolean dbExists() { } CommandLine dbQueryCommand = psqlCommand().addArgument("--command").addArgument(""); - log.debug("Command line: " + dbQueryCommand.toString()); + log.debug("dbQueryCommand={}", dbQueryCommand.toString()); return ExternalProcess.simpleExternalProcess(dbQueryCommand).execute(true) == 0; } @@ -313,4 +321,9 @@ public static void waitForProcessToSettle(long millis) { Thread.currentThread().interrupt(); } } + + @Override + public int maxExpressions() { + return MAX_EXPRESSIONS; + } } diff --git a/src/main/java/gov/nasa/ziggy/services/database/SqlRunner.java b/src/main/java/gov/nasa/ziggy/services/database/SqlRunner.java index 754667a..1d801f9 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/SqlRunner.java +++ b/src/main/java/gov/nasa/ziggy/services/database/SqlRunner.java @@ -69,7 +69,7 @@ public Connection connection() { */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private Connection connect() { - log.info("Connecting to: {}", connectInfo); + log.info("Connecting to {}", connectInfo); try { Class.forName(connectInfo.getDriverName()); cachedConnection = DriverManager.getConnection(connectInfo.getUrl(), @@ -179,7 +179,7 @@ public void close() { cachedConnection.close(); cachedConnection = null; } catch (SQLException e) { - log.warn("Could not close database connection: " + e.getMessage(), e); + log.warn("Could not close database connection: {}", e.getMessage(), e); } } } diff --git a/src/main/java/gov/nasa/ziggy/services/database/ZiggyHibernateConfiguration.java b/src/main/java/gov/nasa/ziggy/services/database/ZiggyHibernateConfiguration.java index 75ee0e0..2f82f09 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/ZiggyHibernateConfiguration.java +++ b/src/main/java/gov/nasa/ziggy/services/database/ZiggyHibernateConfiguration.java @@ -61,7 +61,7 @@ public static Configuration buildHibernateConfiguration() { String key = (String) iter.next(); String value = ziggyConfig.getString(key); - log.debug("copying property, key=" + key + ", value=" + value); + log.debug("copying property, key={}, value={}", key, value); hibernateConfig.setProperty(key, value); } @@ -69,12 +69,12 @@ public static Configuration buildHibernateConfiguration() { hibernateConfig.setProperty(HIBERNATE_DIALECT.property(), hibernateDialect()); hibernateConfig.setProperty(HIBERNATE_DRIVER.property(), driverClassName()); - log.info("Database URL: " + hibernateConfig.getProperty(HIBERNATE_URL.property())); - log.debug("Database User: " + hibernateConfig.getProperty(HIBERNATE_USERNAME.property())); + log.info("Database URL {}", hibernateConfig.getProperty(HIBERNATE_URL.property())); + log.debug("Database user {}", hibernateConfig.getProperty(HIBERNATE_USERNAME.property())); Set> detectedClasses = annotatedClasses(); - log.debug("Adding " + detectedClasses.size() + " annotated POJOs to Hibernate"); + log.debug("Adding {} annotated POJOs to Hibernate", detectedClasses.size()); for (Class clazz : detectedClasses) { hibernateConfig.addAnnotatedClass(clazz); diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java index 977ac1a..0f56cc7 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java @@ -4,6 +4,7 @@ import java.util.Objects; import java.util.Set; +import gov.nasa.ziggy.util.SystemProxy; import jakarta.persistence.ElementCollection; import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; @@ -47,7 +48,7 @@ public ZiggyEvent(String eventHandlerName, String pipelineName, long pipelineIns this.eventHandlerName = eventHandlerName; this.pipelineName = pipelineName; this.eventLabels = eventLabels; - eventTime = new Date(); + eventTime = new Date(SystemProxy.currentTimeMillis()); pipelineInstanceId = pipelineInstance; } @@ -55,50 +56,26 @@ public Long getId() { return id; } - public void setId(Long id) { - this.id = id; - } - public String getEventHandlerName() { return eventHandlerName; } - public void setEventHandlerName(String eventHandlerName) { - this.eventHandlerName = eventHandlerName; - } - public String getPipelineName() { return pipelineName; } - public void setPipelineName(String pipelineName) { - this.pipelineName = pipelineName; - } - public Date getEventTime() { return eventTime; } - public void setEventTime(Date eventTime) { - this.eventTime = eventTime; - } - public long getPipelineInstanceId() { return pipelineInstanceId; } - public void setPipelineInstanceId(long pipelineInstance) { - pipelineInstanceId = pipelineInstance; - } - public Set getEventLabels() { return eventLabels; } - public void setEventLabels(Set eventLabels) { - this.eventLabels = eventLabels; - } - @Override public int hashCode() { return Objects.hash(id); @@ -115,4 +92,11 @@ public boolean equals(Object obj) { ZiggyEvent other = (ZiggyEvent) obj; return Objects.equals(id, other.id); } + + @Override + public String toString() { + return "eventHandlerName=" + eventHandlerName + ", pipelineName=" + pipelineName + + ", eventTime=" + eventTime + ", pipelineInstanceId=" + pipelineInstanceId + + ", eventLabels=" + eventLabels; + } } diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java index dcc56b3..12359ef 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java @@ -26,9 +26,10 @@ import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.messages.EventHandlerToggleStateRequest; import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; @@ -178,8 +179,10 @@ public void run() { // don't want the event handler infrastructure itself to fail as a result. For // this reason, we catch Exception here (to include both runtime exceptions and // unexpected checked exceptions), and then stop this particular event handler. - log.error("ZiggyEventHandler " + name + " disabled due to exception", e); - alertService().generateAndBroadcastAlert("Event handler " + name, 0, Severity.WARNING, + // TODO Use AlertService.DEFAULT_TASK here instead? + log.error("ZiggyEventHandler {} disabled due to exception", name, e); + alertService().generateAndBroadcastAlert("Event handler " + name, + new PipelineTask(null, null, null), Severity.WARNING, "Event handler shut down due to exception"); stop(); } @@ -226,11 +229,10 @@ Set readyFilesForExecution() { */ private void startPipeline(ReadyFile readyFile) { - log.info("Event handler " + name + " detected event"); - log.info( - "Event name \"" + readyFile.getName() + "\" has " + readyFile.labelCount() + " labels"); - log.debug("Event handler labels: " + readyFile.labels()); - log.info("Event handler " + name + " starting pipeline " + pipelineName + "..."); + log.info("Event handler {} detected event", name); + log.info("Event {} has {} labels", readyFile.getName(), readyFile.labelCount()); + log.debug("Event handler labels are {}", readyFile.labels()); + log.info("Event handler {} starting pipeline {}", name, pipelineName); // Create a new pipeline instance that includes the event handler labels. PipelineDefinition pipelineDefinition = pipelineDefinitionOperations() @@ -240,14 +242,14 @@ private void startPipeline(ReadyFile readyFile) { ZiggyMessenger.publish(new InvalidateConsoleModelsMessage()); ziggyEventOperations().newZiggyEvent(name, pipelineName, pipelineInstance.getId(), readyFile.getLabels()); - log.info("Event handler " + name + " starting pipeline " + pipelineName + "...done"); + log.info("Event handler {} starting pipeline {}...done", name, pipelineName); } /** * Toggles the state of the {@link ZiggyEventHandler} from enabled to disabled or vice-versa. */ public void toggleStatus() { - log.debug("Toggling state of event handler \"" + name + "\""); + log.debug("Toggling state of event handler {}", name); if (watcherThread == null) { // indicates disabled start(); } else { diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java index 7516cb3..c3af7f7 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java @@ -85,28 +85,28 @@ public void importFromFiles() { for (File file : files) { ZiggyEventHandlerFile handlerFile = null; try { - log.info("Unmarshaling file " + file.getName() + "..."); + log.info("Unmarshaling file {}", file.getName()); handlerFile = xmlManager.unmarshal(file); } catch (Exception e) { - log.error("Unable to unmarshal file " + file.getName(), e); + log.error("Unable to unmarshal file {}", file.getName(), e); continue; } - log.info("Unmarshaling file " + file.getName() + "...done"); + log.info("Unmarshaling file {}...done", file.getName()); for (ZiggyEventHandler handler : handlerFile.getZiggyEventHandlers()) { if (!pipelineDefinitionNames.contains(handler.getPipelineName())) { - log.error("Handler " + handler.getName() + " fires pipeline " - + handler.getPipelineName() + ", which is not defined"); - log.error("Not persisting handler " + handler.getName()); + log.error("Handler {} fires pipeline {}, which is not defined", + handler.getName(), handler.getPipelineName()); + log.error("Not persisting handler {}", handler.getName()); continue; } try { - log.info("Persisting handler " + handler.getName() + "..."); + log.info("Persisting handler {}", handler.getName()); ziggyEventOperations().mergeEventHandler(handler); } catch (Exception e) { - log.error("Unable to persist handler " + handler.getName(), e); + log.error("Unable to persist handler {}", handler.getName(), e); continue; } - log.info("Persisting handler " + handler.getName() + "...done"); + log.info("Persisting handler {}...done", handler.getName()); } } log.info("Done importing Ziggy event handlers"); diff --git a/src/main/java/gov/nasa/ziggy/services/logging/TaskLog.java b/src/main/java/gov/nasa/ziggy/services/logging/TaskLog.java index 23a81e7..8b90fa3 100644 --- a/src/main/java/gov/nasa/ziggy/services/logging/TaskLog.java +++ b/src/main/java/gov/nasa/ziggy/services/logging/TaskLog.java @@ -29,6 +29,7 @@ import gov.nasa.ziggy.module.LocalAlgorithmExecutor; import gov.nasa.ziggy.module.remote.Qsub; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; @@ -98,14 +99,16 @@ public FileAppender taskLogFileAppender(Path taskLog) { public static final String CLI_APPENDER_NAME = "cli"; + private static PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + /** * Locates all the log files for a given pipeline task. */ - public static Set searchForLogFiles(long taskId) { + public static Set searchForLogFiles(PipelineTask pipelineTask) { Set taskLogInfoSet = new TreeSet<>(); for (LogType logType : LogType.values()) { - for (File taskLogFile : searchForLogFilesOfType(logType, taskId)) { + for (File taskLogFile : searchForLogFilesOfType(logType, pipelineTask)) { taskLogInfoSet.add(new TaskLogInformation(taskLogFile)); } } @@ -127,14 +130,14 @@ public static String algorithmLogFileSystemProperty(PipelineTask pipelineTask, i return "-D" + PropertyName.ZIGGY_LOG_FILE.property() + "=" + DirectoryProperties.algorithmLogsDir() .toAbsolutePath() - .resolve(pipelineTask.logFilename(jobIndex)) + .resolve(pipelineTaskDataOperations.logFilename(pipelineTask, jobIndex)) .toString(); } /** Returns the ziggy.algorithmName Java property for use with {@link ComputeNodeMaster}. */ public static String algorithmNameSystemProperty(PipelineTask pipelineTask) { return "-D" + PropertyName.ZIGGY_ALGORITHM_NAME.property() + "=" - + pipelineTask.getModuleName(); + + pipelineTask.getExecutableName(); } public static String ziggyLogFileSystemProperty(PipelineTask pipelineTask) { @@ -146,23 +149,24 @@ public static String ziggyLogFileSystemProperty(PipelineTask pipelineTask) { return "-D" + PropertyName.ZIGGY_LOG_FILE.property() + "=" + DirectoryProperties.taskLogDir() .toAbsolutePath() - .resolve(pipelineTask.logFilename(LOCAL_LOG_FILE_JOB_INDEX)) + .resolve( + pipelineTaskDataOperations.logFilename(pipelineTask, LOCAL_LOG_FILE_JOB_INDEX)) .toString(); } /** * Locates all the log files of a given type for a given pipeline task. */ - private static File[] searchForLogFilesOfType(LogType logType, long taskId) { - log.debug("Searching directory " + logType.logDir().toString() + " for log files"); + private static File[] searchForLogFilesOfType(LogType logType, PipelineTask pipelineTask) { + log.debug("Searching directory {} for log files", logType.logDir().toString()); if (!Files.exists(logType.logDir()) || !Files.isDirectory(logType.logDir())) { return new File[0]; } return logType.logDir().toFile().listFiles((FilenameFilter) (dir, name) -> { Matcher matcher = TaskLogInformation.LOG_FILE_NAME_PATTERN.matcher(name); if (matcher.matches()) { - return Long - .parseLong(matcher.group(TaskLogInformation.TASK_ID_GROUP_NUMBER)) == taskId; + return Long.parseLong( + matcher.group(TaskLogInformation.TASK_ID_GROUP_NUMBER)) == pipelineTask.getId(); } return false; }); @@ -205,4 +209,9 @@ private static FileAppender.Builder fileAppenderBuilder(File taskLogFile) { .setName(taskLogFile.getName()) .setLayout(cliLayout()); } + + static void setPipelineTaskDataOperations( + PipelineTaskDataOperations pipelineTaskDataOperations) { + TaskLog.pipelineTaskDataOperations = pipelineTaskDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/AllJobsFinishedMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/AllJobsFinishedMessage.java new file mode 100644 index 0000000..8d977a9 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/AllJobsFinishedMessage.java @@ -0,0 +1,28 @@ +package gov.nasa.ziggy.services.messages; + +import gov.nasa.ziggy.module.AlgorithmMonitor; +import gov.nasa.ziggy.module.TaskMonitor; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +/** + * Message from the {@link AlgorithmMonitor} that notifies the {@link TaskMonitor} that all the + * remote jobs for the task have finished. This is needed because remote jobs can exit silently, in + * which case the task monitor doesn't know that nobody is processing task data any longer. + * + * @author PT + * @author Bill Wohler + */ +public class AllJobsFinishedMessage extends PipelineMessage { + + private static final long serialVersionUID = 20240909L; + + private final PipelineTask pipelineTask; + + public AllJobsFinishedMessage(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + public PipelineTask getPipelineTask() { + return pipelineTask; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java index 7e71779..7d4f75c 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java @@ -25,7 +25,7 @@ public class EventHandlerToggleStateRequest extends PipelineMessage { * private, which in turn forces the user to verify privileges before the request is sent. */ public static void requestEventHandlerToggle(String handlerName) { - log.debug("Sending toggle request for event handler \"" + handlerName + "\""); + log.debug("Sending toggle request for event handler {}", handlerName); ZiggyMessenger.publish(new EventHandlerToggleStateRequest(handlerName)); } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/HaltTasksRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/HaltTasksRequest.java new file mode 100644 index 0000000..c10d0a5 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/HaltTasksRequest.java @@ -0,0 +1,30 @@ +package gov.nasa.ziggy.services.messages; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +/** + * Requests that the supervisor halt the selected tasks if they are waiting in the worker queue to + * be processed, and then makes the same request of worker processes that are processing any of + * these tasks. processes. The tasks are specified by task ID. + * + * @author PT + * @author Bill Wohler + */ +public class HaltTasksRequest extends PipelineMessage { + + private static final long serialVersionUID = 20240913L; + + private final List pipelineTasks; + + public HaltTasksRequest(Collection pipelineTasks) { + this.pipelineTasks = new ArrayList<>(pipelineTasks); + } + + public List getPipelineTasks() { + return pipelineTasks; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/KillTasksRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/KillTasksRequest.java deleted file mode 100644 index c1e7043..0000000 --- a/src/main/java/gov/nasa/ziggy/services/messages/KillTasksRequest.java +++ /dev/null @@ -1,56 +0,0 @@ -package gov.nasa.ziggy.services.messages; - -import java.util.ArrayList; -import java.util.Collection; -import java.util.List; - -import gov.nasa.ziggy.module.StateFile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.util.Requestor; - -/** - * Requests that the supervisor kill the selected tasks if they are waiting in the worker queue to - * be processed, and then makes the same request of worker processes that are processing any of - * these tasks. processes. The tasks are specified by task ID. - * - * @author PT - */ -public class KillTasksRequest extends SpecifiedRequestorMessage { - - private static final long serialVersionUID = 20231030L; - - private final List taskIds; - - private KillTasksRequest(Requestor requestor, List taskIds) { - super(requestor); - this.taskIds = taskIds; - } - - public static KillTasksRequest forTaskIds(Requestor requestor, Collection taskIds) { - return new KillTasksRequest(requestor, new ArrayList<>(taskIds)); - } - - public static KillTasksRequest forPipelineTasks(Requestor requestor, - Collection pipelineTasks) { - - List taskIds = new ArrayList<>(); - for (PipelineTask pipelineTask : pipelineTasks) { - taskIds.add(pipelineTask.getId()); - } - return new KillTasksRequest(requestor, taskIds); - } - - public static KillTasksRequest forStateFiles(Requestor requestor, - Collection stateFiles) { - - List taskIds = new ArrayList<>(); - for (StateFile stateFile : stateFiles) { - taskIds.add(stateFile.getPipelineTaskId()); - } - return new KillTasksRequest(requestor, taskIds); - } - - public List getTaskIds() { - return taskIds; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/KilledTaskMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/KilledTaskMessage.java deleted file mode 100644 index 07a9180..0000000 --- a/src/main/java/gov/nasa/ziggy/services/messages/KilledTaskMessage.java +++ /dev/null @@ -1,25 +0,0 @@ -package gov.nasa.ziggy.services.messages; - -import gov.nasa.ziggy.util.Requestor; - -/** - * Response sent in reply to a {@link KillTasksRequest} message. The reply tells the sender that a - * particular task has been killed (or at least that the task-killing activities completed without - * an exception). - * - * @author PT - */ -public class KilledTaskMessage extends SpecifiedRequestorMessage { - private static final long serialVersionUID = 20231030L; - - private final long taskId; - - public KilledTaskMessage(Requestor requestor, long taskId) { - super(requestor); - this.taskId = taskId; - } - - public long getTaskId() { - return taskId; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/MonitorAlgorithmRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/MonitorAlgorithmRequest.java index b4fb080..6d0ab59 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/MonitorAlgorithmRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/MonitorAlgorithmRequest.java @@ -1,34 +1,62 @@ package gov.nasa.ziggy.services.messages; -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; + import gov.nasa.ziggy.module.AlgorithmMonitor; -import gov.nasa.ziggy.module.StateFile; +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.module.remote.RemoteJobInformation; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.supervisor.PipelineSupervisor; import gov.nasa.ziggy.worker.PipelineWorker; /** * Message sent from the {@link PipelineWorker} to the {@link PipelineSupervisor} to request that a - * given {@link StateFile} be added to either the local or remote {@link AlgorithmMonitor}. + * given task be added to either the local or remote {@link AlgorithmMonitor}. * * @author PT + * @author Bill Wohler */ public class MonitorAlgorithmRequest extends PipelineMessage { - private static final long serialVersionUID = 20230511L; + private static final long serialVersionUID = 20240909L; + private final PipelineTask pipelineTask; + private final List remoteJobsInformation; + private final String taskDir; private final AlgorithmType algorithmType; - private final StateFile stateFile; - public MonitorAlgorithmRequest(StateFile stateFile, AlgorithmType algorithmType) { + public MonitorAlgorithmRequest(PipelineTask pipelineTask, Path taskDir) { + this(pipelineTask, taskDir, AlgorithmType.LOCAL, null); + } + + public MonitorAlgorithmRequest(PipelineTask pipelineTask, Path taskDir, + List remoteJobInformation) { + this(pipelineTask, taskDir, AlgorithmType.REMOTE, remoteJobInformation); + } + + private MonitorAlgorithmRequest(PipelineTask pipelineTask, Path taskDir, + AlgorithmType algorithmType, List remoteJobsInformation) { + this.pipelineTask = pipelineTask; + this.remoteJobsInformation = remoteJobsInformation; + this.taskDir = taskDir.toString(); this.algorithmType = algorithmType; - this.stateFile = stateFile; + } + + public PipelineTask getPipelineTask() { + return pipelineTask; + } + + public Path getTaskDir() { + return Paths.get(taskDir); } public AlgorithmType getAlgorithmType() { return algorithmType; } - public StateFile getStateFile() { - return stateFile; + public List getRemoteJobsInformation() { + return remoteJobsInformation; } } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/RemoveTaskFromKilledTasksMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/RemoveTaskFromKilledTasksMessage.java index 6e4b156..0eb8c3c 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/RemoveTaskFromKilledTasksMessage.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/RemoveTaskFromKilledTasksMessage.java @@ -1,5 +1,6 @@ package gov.nasa.ziggy.services.messages; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.supervisor.PipelineSupervisor; import gov.nasa.ziggy.supervisor.TaskRequestHandlerLifecycleManager; @@ -9,18 +10,19 @@ * killed previously, take it out of that list. * * @author PT + * @author Bill Wohler */ public class RemoveTaskFromKilledTasksMessage extends PipelineMessage { - private static final long serialVersionUID = 20231018L; + private static final long serialVersionUID = 20240905L; - private final long taskId; + private final PipelineTask pipelineTask; - public RemoveTaskFromKilledTasksMessage(long taskId) { - this.taskId = taskId; + public RemoveTaskFromKilledTasksMessage(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; } - public long getTaskId() { - return taskId; + public PipelineTask getPipelineTask() { + return pipelineTask; } } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/RestartTasksRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/RestartTasksRequest.java index 9f2c05e..c407b25 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/RestartTasksRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/RestartTasksRequest.java @@ -1,8 +1,6 @@ package gov.nasa.ziggy.services.messages; -import java.util.ArrayList; import java.util.List; -import java.util.stream.Collectors; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; @@ -11,24 +9,25 @@ * Notifies the supervisor that pipeline tasks should be restarted. * * @author PT + * @author Bill Wohler */ public class RestartTasksRequest extends PipelineMessage { - private static final long serialVersionUID = 20240423L; - private final List pipelineTaskIds = new ArrayList<>(); + private static final long serialVersionUID = 20240909L; + + private final List pipelineTasks; private final boolean doTransitionOnly; private final RunMode runMode; public RestartTasksRequest(List pipelineTasks, boolean doTransitionOnly, RunMode runMode) { + this.pipelineTasks = pipelineTasks; this.doTransitionOnly = doTransitionOnly; this.runMode = runMode; - pipelineTaskIds - .addAll(pipelineTasks.stream().map(PipelineTask::getId).collect(Collectors.toSet())); } - public List getPipelineTaskIds() { - return pipelineTaskIds; + public List getPipelineTasks() { + return pipelineTasks; } public boolean isDoTransitionOnly() { diff --git a/src/main/java/gov/nasa/ziggy/services/messages/TaskHaltedMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/TaskHaltedMessage.java new file mode 100644 index 0000000..de5d65a --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/TaskHaltedMessage.java @@ -0,0 +1,24 @@ +package gov.nasa.ziggy.services.messages; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +/** + * Notifies subscribers that a given {@link PipelineTask} has successfully halted. + * + * @author PT + * @author Bill Wohler + */ +public class TaskHaltedMessage extends PipelineMessage { + + private static final long serialVersionUID = 20240913L; + + private final PipelineTask pipelineTask; + + public TaskHaltedMessage(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + public PipelineTask getPipelineTask() { + return pipelineTask; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java index 4aadd2b..2105a0b 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java @@ -1,10 +1,6 @@ package gov.nasa.ziggy.services.messages; -import java.util.Set; - import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.logging.TaskLogInformation; -import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.util.Requestor; /** @@ -12,30 +8,20 @@ * * @see gov.nasa.ziggy.services.logging.TaskLogInformation * @author PT + * @author Bill Wohler */ public final class TaskLogInformationRequest extends SpecifiedRequestorMessage { - private static final long serialVersionUID = 20230614L; + private static final long serialVersionUID = 20240909L; - private final long taskId; + private final PipelineTask pipelineTask; - private TaskLogInformationRequest(Requestor sender, long taskId) { + public TaskLogInformationRequest(Requestor sender, PipelineTask pipelineTask) { super(sender); - this.taskId = taskId; - } - - /** - * Performs the request for task log information and returns the information for all logs as a - * {@link Set} of {@link TaskLogInformation} instances. This method is the only public method to - * retrieve a task log, as the {@link TaskLogInformationRequest} constructor is private. This - * prevents callers from attempting to bypass the privilege verification performed in this - * method. - */ - public static void requestTaskLogInformation(Requestor sender, PipelineTask task) { - ZiggyMessenger.publish(new TaskLogInformationRequest(sender, task.getId())); + this.pipelineTask = pipelineTask; } - public long getTaskId() { - return taskId; + public PipelineTask getPipelineTask() { + return pipelineTask; } } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/TaskProcessingCompleteMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/TaskProcessingCompleteMessage.java new file mode 100644 index 0000000..8f7801e --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/TaskProcessingCompleteMessage.java @@ -0,0 +1,18 @@ +package gov.nasa.ziggy.services.messages; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +public class TaskProcessingCompleteMessage extends PipelineMessage { + + private static final long serialVersionUID = 20240909L; + + private final PipelineTask pipelineTask; + + public TaskProcessingCompleteMessage(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + public PipelineTask getPipelineTask() { + return pipelineTask; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/TaskRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/TaskRequest.java index 4ab5859..c532564 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/TaskRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/TaskRequest.java @@ -8,20 +8,22 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** * Requests that the supervisor add a new task to its task queue. * * @author Todd Klaus * @author PT + * @author Bill Wohler */ public class TaskRequest extends PipelineMessage implements Comparable { - private static final long serialVersionUID = 20230511L; + private static final long serialVersionUID = 20240909L; private final long instanceId; private final long instanceNodeId; private final long pipelineDefinitionNodeId; - private final long taskId; + private final PipelineTask pipelineTask; private final Priority priority; private final RunMode runMode; @@ -32,11 +34,11 @@ public class TaskRequest extends PipelineMessage implements Comparable { - private static final long serialVersionUID = 20230522L; + private static final long serialVersionUID = 20240909L; public static Logger log = LoggerFactory.getLogger(WorkerStatusMessage.class); private final int workerNumber; private final String state; private final String instanceId; - private final String taskId; + private final PipelineTask pipelineTask; private final String module; private final String moduleUow; private final long processingStartTime; private final boolean lastMessageFromWorker; - public WorkerStatusMessage(int workerNumber, String state, String instanceId, String taskId, - String module, String moduleUow, long processingStartTime, boolean lastMessageFromWorker) { + // Cached value of database lookup. + + public WorkerStatusMessage(int workerNumber, String state, String instanceId, + PipelineTask pipelineTask, String module, String moduleUow, long processingStartTime, + boolean lastMessageFromWorker) { this.workerNumber = workerNumber; this.state = state; this.instanceId = instanceId; - this.taskId = taskId; + this.pipelineTask = pipelineTask; this.module = module; this.moduleUow = moduleUow; this.processingStartTime = processingStartTime; @@ -67,12 +72,12 @@ public int getWorkerNumber() { @Override public String briefStatus() { return "WS:" + super.briefStatus() + "(" + workerNumber + "):" + "[" + state + ":" - + instanceId + "-" + taskId + ":" + module + "{" + moduleUow + "}:since " + + instanceId + "-" + pipelineTask.getId() + ":" + module + "{" + moduleUow + "}:since " + new Date(processingStartTime) + "]"; } - public String getTaskId() { - return taskId; + public PipelineTask getPipelineTask() { + return pipelineTask; } public String getInstanceId() { @@ -87,7 +92,7 @@ public boolean isLastMessageFromWorker() { public int hashCode() { final int prime = 31; int result = super.hashCode(); - return prime * result + Objects.hash(taskId); + return prime * result + Objects.hash(pipelineTask); } @Override @@ -99,11 +104,11 @@ public boolean equals(Object obj) { return false; } WorkerStatusMessage other = (WorkerStatusMessage) obj; - return Objects.equals(taskId, other.taskId); + return Objects.equals(pipelineTask, other.pipelineTask); } @Override public int compareTo(WorkerStatusMessage o) { - return (int) (Long.parseLong(taskId) - Long.parseLong(o.taskId)); + return pipelineTask.compareTo(o.pipelineTask); } } diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java b/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java index b6a99ce..c2525a9 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java @@ -103,9 +103,11 @@ void start() { @AcceptableCatchBlock(rationale = Rationale.CLEANUP_BEFORE_EXIT) private void initializeHeartbeatManagerInternal() { + log.debug("Initializing heartbeat manager"); + heartbeatCountdownLatch = new CountDownLatch(1); SystemProxy.currentTimeMillis(); - log.debug("Heartbeat time: " + heartbeatTime); + log.debug("Heartbeat time is {}", heartbeatTime); try { // If the console, for example, is stopped for a few hours and then resumed, // checkForHeartbeat() will call initializeHeartbeatManager() a lot, which can result in @@ -134,7 +136,7 @@ private void initializeHeartbeatManagerInternal() { startHeartbeatListener(); } started = true; - log.debug("initializeHeartbeatManagerInternal: done"); + log.debug("Initializing heartbeat manager...done"); } /** @@ -163,7 +165,7 @@ protected void startHeartbeatListener() { ZiggyShutdownHook.addShutdownHook(() -> { heartbeatListener.shutdownNow(); }); - log.info("Heartbeat listener thread started"); + log.info("Starting heartbeat listener thread...done"); } /** diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java index 3650e8b..4be9755 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java @@ -9,7 +9,7 @@ import java.util.concurrent.CountDownLatch; import java.util.concurrent.LinkedBlockingQueue; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java index 816a62a..0b49223 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java @@ -22,6 +22,7 @@ import gov.nasa.ziggy.ui.ZiggyConsole; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.ZiggyUtils; import gov.nasa.ziggy.util.os.ProcessUtils; import gov.nasa.ziggy.worker.PipelineWorker; @@ -50,7 +51,7 @@ public class ZiggyRmiClient implements ZiggyRmiClientService { private static final String RMI_REGISTRY_HOST = "localhost"; private static final int REGISTRY_LOOKUP_EFFORTS = 25; - private static final int REGISTRY_LOOKUP_PAUSE_MILLIS = 200; + private static final long REGISTRY_LOOKUP_PAUSE_MILLIS = 200; /** * Singleton instance of {@link ZiggyRmiClient} class. All threads in the UI process can access @@ -80,26 +81,9 @@ private ZiggyRmiClient(String clientType) throws RemoteException, NotBoundExcept // Get the stub that the server provided. In case the server just started, try every 200 ms // for five seconds. - ZiggyRmiServerService service = null; - for (int i = 0; i < REGISTRY_LOOKUP_EFFORTS; i++) { - log.info("Looking up services in registry (take {}/{})", i + 1, - REGISTRY_LOOKUP_EFFORTS); - try { - service = (ZiggyRmiServerService) registry - .lookup(ZiggyRmiServerService.SERVICE_NAME); - break; - } catch (RemoteException | NotBoundException e) { - if (i == REGISTRY_LOOKUP_EFFORTS - 1) { - throw e; - } - try { - Thread.sleep(REGISTRY_LOOKUP_PAUSE_MILLIS); - } catch (InterruptedException interrupt) { - Thread.currentThread().interrupt(); - } - } - } - ziggyRmiServerService = service; + ziggyRmiServerService = ZiggyUtils.tryPatiently("Looking up services in registry", + REGISTRY_LOOKUP_EFFORTS, REGISTRY_LOOKUP_PAUSE_MILLIS, + () -> (ZiggyRmiServerService) registry.lookup(ZiggyRmiServerService.SERVICE_NAME)); } /** diff --git a/src/main/java/gov/nasa/ziggy/services/metrics/MetricsReaperThread.java b/src/main/java/gov/nasa/ziggy/services/metrics/MetricsReaperThread.java index ca2410b..b70e744 100644 --- a/src/main/java/gov/nasa/ziggy/services/metrics/MetricsReaperThread.java +++ b/src/main/java/gov/nasa/ziggy/services/metrics/MetricsReaperThread.java @@ -33,32 +33,28 @@ public MetricsReaperThread() { @AcceptableCatchBlock(rationale = Rationale.SYSTEM_EXIT) public void run() { try { - log.info("MetricsReaperThread: STARTED"); + log.info("Starting MetricsReaperThread"); ImmutableConfiguration config = ZiggyConfiguration.getInstance(); checkIntervalMillis = config.getInt(CHECK_INTERVAL_MINS_PROP, DEFAULT_CHECK_INTERVAL_MINS) * 60 * 1000; maxRows = config.getInt(MAX_ROWS_PROP, DEFAULT_MAX_ROWS); - log.info("MetricsReaperThread: maxRows = " + maxRows); - log.info("MetricsReaperThread: checkIntervalMillis = " + checkIntervalMillis); + log.info("maxRows={}, checkIntervalMillis={}", maxRows, checkIntervalMillis); while (true) { long now = System.currentTimeMillis(); if (now - lastCheck > checkIntervalMillis) { - log.info("MetricsReaperThread: woke up to check rowCount"); - + log.info("Checking rowCount"); metricsOperations().deleteOldMetrics(maxRows); - lastCheck = now; - - log.info("MetricsReaperThread: check complete"); + log.info("Checking rowCount...done"); } Thread.sleep(1000); } } catch (Throwable t) { - log.error("MetricsReaperThread: caught: ", t); + log.error("Caught exception", t); System.exit(-1); } } diff --git a/src/main/java/gov/nasa/ziggy/services/process/AbstractPipelineProcess.java b/src/main/java/gov/nasa/ziggy/services/process/AbstractPipelineProcess.java index 311406e..6f800c5 100644 --- a/src/main/java/gov/nasa/ziggy/services/process/AbstractPipelineProcess.java +++ b/src/main/java/gov/nasa/ziggy/services/process/AbstractPipelineProcess.java @@ -9,6 +9,7 @@ import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.database.DatabaseService; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.os.ProcessUtils; /** @@ -59,8 +60,7 @@ protected void initialize() { ImmutableConfiguration config = ZiggyConfiguration.getInstance(); - log.info("Starting process {} ({})", processInfo, - config.getString(PropertyName.ZIGGY_VERSION.property())); + log.info("Starting process {} ({})", processInfo, BuildInfo.ziggyVersion()); ZiggyConfiguration.logJvmProperties(); if (initDatabaseService) { diff --git a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java index 4300eaf..429edc2 100644 --- a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java +++ b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java @@ -528,12 +528,12 @@ private synchronized static void setExternalProcessesShutdownHook() { ZiggyShutdownHook.addShutdownHook(() -> { pool.shutdown(); childProcessIds.addAll(childProcessIdsPreviousCall); - log.debug("SHUTDOWN: Sending SIGTERM to child processes"); + log.debug("Sending SIGTERM to child processes"); for (Long processId : childProcessIds) { - log.debug("SHUTDOWN: Sending SIGTERM to PID " + processId); + log.debug("Sending SIGTERM to PID {}", processId); ProcessUtils.sendSigtermToProcess(processId); } - log.debug("SHUTDOWN: done with SIGTERMs"); + log.debug("Sending SIGTERM to child processes...done"); }); externalProcessesShutdownHookSet = true; } diff --git a/src/main/java/gov/nasa/ziggy/services/process/StatusMessageBroadcaster.java b/src/main/java/gov/nasa/ziggy/services/process/StatusMessageBroadcaster.java index 9d100fe..8c87fcb 100644 --- a/src/main/java/gov/nasa/ziggy/services/process/StatusMessageBroadcaster.java +++ b/src/main/java/gov/nasa/ziggy/services/process/StatusMessageBroadcaster.java @@ -34,7 +34,7 @@ public StatusMessageBroadcaster(ProcessInfo processInfo) { } public synchronized void addStatusReporter(final StatusReporter reporter) { - log.info("Adding a status reporter of class " + reporter.getClass().getName()); + log.info("Adding a status reporter of class {}", reporter.getClass().getName()); reporters.add(reporter); } diff --git a/src/main/java/gov/nasa/ziggy/supervisor/InstanceReporter.java b/src/main/java/gov/nasa/ziggy/supervisor/InstanceReporter.java index 2bdc921..5fdcca7 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/InstanceReporter.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/InstanceReporter.java @@ -9,8 +9,9 @@ import java.util.List; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -27,7 +28,8 @@ */ public class InstanceReporter { - private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) public File report(PipelineInstance instance, File outputDir) { @@ -39,14 +41,16 @@ public File report(PipelineInstance instance, File outputDir) { File reportFile = new File(outputDir, "instance-" + instance.getId() + "-report.txt"); reportFile.delete(); - try (PrintStream printStream = new PrintStream(reportFile, ZiggyFileUtils.ZIGGY_CHARSET_NAME)) { + try (PrintStream printStream = new PrintStream(reportFile, + ZiggyFileUtils.ZIGGY_CHARSET_NAME)) { printStream.print("state: " + instance.getState() + "\n\n"); InstancesDisplayModel instancesDisplayModel = new InstancesDisplayModel(instance); instancesDisplayModel.print(printStream, "Instance Summary"); printStream.println(); - List tasks = pipelineTaskOperations().pipelineTasks(instance); + List tasks = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(instance); TaskSummaryDisplayModel taskSummaryDisplayModel = new TaskSummaryDisplayModel( new TaskCounts(tasks)); @@ -75,4 +79,8 @@ public File report(PipelineInstance instance, File outputDir) { PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java b/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java index a82c22c..728ff32 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java @@ -79,18 +79,9 @@ void initialize(StartPipelineRequest triggerRequest) { } String startNodeName = triggerRequest.getStartNodeName(); String endNodeName = triggerRequest.getEndNodeName(); - String startString, endString; - if (!StringUtils.isBlank(startNodeName)) { - startString = "start node: " + startNodeName; - } else { - startString = "start node not set"; - } - if (!StringUtils.isBlank(endNodeName)) { - endString = "end node: " + endNodeName; - } else { - endString = "end node not set"; - } - log.info("PipelineInstanceManager.initialize: " + startString + ", " + endString); + log.info("start node is {}, end node is {}", + StringUtils.isBlank(startNodeName) ? "not set" : startNodeName, + StringUtils.isBlank(endNodeName) ? "not set" : endNodeName); repeatIntervalMillis = triggerRequest.getRepeatIntervalSeconds() * 1000; pipeline = pipelineDefinitionOperations() .lockAndReturnLatestPipelineDefinition(triggerRequest.getPipelineName()); @@ -205,19 +196,13 @@ private boolean waitAndCheckStatus() throws InterruptedException { loopStatus.setInstanceCompleted(false); } if (loopStatus.keepLooping()) { - log.info("Current pipeline instance " + currentInstanceId - + " still running, next instance start delayed"); + log.info("Current pipeline instance {} still running, next instance start delayed", + currentInstanceId); Thread.sleep(checkAgainIntervalMillis); } else { - String loopMessage; - if (loopStatus.instanceCompleted()) { - loopMessage = "Current pipeline instance " + currentInstanceId - + " completed, next instance starting"; - } else { - loopMessage = "Current pipeline instance " + currentInstanceId - + " errored, next instance canceled"; - } - log.info(loopMessage); + log.info("Current pipeline instance {} {}, next instance {}", currentInstanceId, + loopStatus.instanceCompleted() ? "completed" : "errored", + loopStatus.instanceCompleted() ? "starting" : "canceled"); } } return loopStatus.instanceCompleted(); diff --git a/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java b/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java index cea538a..20c4f8c 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java @@ -15,6 +15,7 @@ import java.nio.file.Path; import java.util.Collection; import java.util.List; +import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ScheduledExecutorService; @@ -31,14 +32,15 @@ import gov.nasa.ziggy.metrics.MetricsDumper; import gov.nasa.ziggy.metrics.report.Memdrone; import gov.nasa.ziggy.module.AlgorithmMonitor; -import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.QueueCommandManager; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; @@ -48,17 +50,18 @@ import gov.nasa.ziggy.services.events.ZiggyEventOperations; import gov.nasa.ziggy.services.logging.TaskLog; import gov.nasa.ziggy.services.messages.EventHandlerRequest; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.services.messages.KilledTaskMessage; import gov.nasa.ziggy.services.messages.RemoveTaskFromKilledTasksMessage; import gov.nasa.ziggy.services.messages.RestartTasksRequest; import gov.nasa.ziggy.services.messages.SingleTaskLogMessage; import gov.nasa.ziggy.services.messages.SingleTaskLogRequest; import gov.nasa.ziggy.services.messages.StartMemdroneRequest; +import gov.nasa.ziggy.services.messages.TaskHaltedMessage; import gov.nasa.ziggy.services.messages.TaskLogInformationMessage; import gov.nasa.ziggy.services.messages.TaskLogInformationRequest; +import gov.nasa.ziggy.services.messages.UpdateProcessingStepMessage; import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; import gov.nasa.ziggy.services.messages.ZiggyEventHandlerInfoMessage; @@ -81,6 +84,7 @@ * * @author Todd Klaus * @author PT + * @author Bill Wohler */ public class PipelineSupervisor extends AbstractPipelineProcess { @@ -93,27 +97,29 @@ public class PipelineSupervisor extends AbstractPipelineProcess { private static final Set ziggyEventHandlers = ConcurrentHashMap.newKeySet(); - // Global list of task IDs that have been killed by the TaskRequestHandlerLifecycleManager, + // Global list of tasks that have been killed by the TaskRequestHandlerLifecycleManager, // a PipelineWorker, or the delete command of a remote system's batch scheduler. - private static final Set killedTaskIds = ConcurrentHashMap.newKeySet(); + private static final Set haltedTasks = ConcurrentHashMap.newKeySet(); private static WorkerResources defaultResources; private ScheduledExecutorService heartbeatExecutor; - private QueueCommandManager queueCommandManager; + private QueueCommandManager queueCommandManager = QueueCommandManager.newInstance(); private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); private PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); private PipelineExecutor pipelineExecutor = new PipelineExecutor(); private PipelineDefinitionOperations pipelineDefinitionOperations = new PipelineDefinitionOperations(); private ZiggyEventOperations ziggyEventOperations = new ZiggyEventOperations(); + private AlgorithmMonitor algorithmMonitor; public PipelineSupervisor(int workerCount, int workerHeapSize) { super(NAME); checkArgument(workerCount > 0, "Worker count must be positive"); checkArgument(workerHeapSize > 0, "Worker heap size must be positive"); defaultResources = new WorkerResources(workerCount, workerHeapSize); - log.debug("Starting pipeline supervisor with " + workerCount + " workers and " - + workerHeapSize + " MB max heap"); + log.debug("Starting pipeline supervisor with {} workers and {} MB max heap", workerCount, + workerHeapSize); } public PipelineSupervisor(boolean messaging, boolean database) { @@ -138,7 +144,7 @@ public void initialize() { H5.loadH5Lib(); - log.info("Subscribing to messages..."); + log.info("Subscribing to messages"); subscribe(); StartPipelineRequestManager.start(); log.info("Subscribing to messages...done"); @@ -162,27 +168,30 @@ public void initialize() { HeartbeatMessage.heartbeatIntervalMillis(), TimeUnit.MILLISECONDS); } ZiggyShutdownHook.addShutdownHook(() -> { - log.info("Shutting down heartbeat executor..."); + log.info("Shutting down heartbeat executor"); heartbeatExecutor.shutdownNow(); log.info("Shutting down heartbeat executor...done"); }); + // Start the algorithm monitor. + log.info("Starting algorithm monitor"); + algorithmMonitor = new AlgorithmMonitor(); + log.info("starting algorithm monitor...done"); + // Start the task request handler lifecycle manager. - log.info("Starting task request handler lifecycle manager..."); + log.info("Starting task request handler lifecycle manager"); TaskRequestHandlerLifecycleManager.initializeInstance(); log.info("Starting task request handler lifecycle manager...done"); - log.info("Starting metrics dumper thread..."); + log.info("Starting metrics dumper thread"); MetricsDumper metricsDumper = new MetricsDumper( AbstractPipelineProcess.getProcessInfo().getPid()); Thread metricsDumperThread = new Thread(metricsDumper, "MetricsDumper"); metricsDumperThread.setDaemon(true); metricsDumperThread.start(); + log.info("Starting metrics dumper thread...done"); - log.info("Starting algorithm monitor threads..."); - AlgorithmMonitor.initialize(); - - log.info("Loading event handlers..."); + log.info("Loading event handlers"); ziggyEventHandlers.addAll(ziggyEventOperations().eventHandlers()); for (ZiggyEventHandler handler : ziggyEventHandlers) { if (handler.isEnableOnClusterStart()) { @@ -232,7 +241,7 @@ private void subscribe() { }); ZiggyMessenger.subscribe(TaskLogInformationRequest.class, message -> { ZiggyMessenger.publish(new TaskLogInformationMessage(message, - TaskLog.searchForLogFiles(message.getTaskId()))); + TaskLog.searchForLogFiles(message.getPipelineTask()))); }); ZiggyMessenger.subscribe(SingleTaskLogRequest.class, message -> { ZiggyMessenger.publish(new SingleTaskLogMessage(message, taskLogContents(message))); @@ -240,31 +249,34 @@ private void subscribe() { ZiggyMessenger.subscribe(WorkerResourcesRequest.class, message -> { ZiggyMessenger.publish(new WorkerResourcesMessage(defaultResources, null)); }); - ZiggyMessenger.subscribe(KillTasksRequest.class, message -> { - killRemoteTasks(message); + ZiggyMessenger.subscribe(HaltTasksRequest.class, message -> { + haltRemoteTasks(message); }); ZiggyMessenger.subscribe(RemoveTaskFromKilledTasksMessage.class, message -> { - killedTaskIds.remove(message.getTaskId()); + haltedTasks.remove(message.getPipelineTask()); }); - ZiggyMessenger.subscribe(KilledTaskMessage.class, message -> { - handleKilledTaskMessage(message); + ZiggyMessenger.subscribe(TaskHaltedMessage.class, message -> { + handleTaskHaltedMessage(message); }); ZiggyMessenger.subscribe(RestartTasksRequest.class, message -> { restartTasks(message); }); + ZiggyMessenger.subscribe(UpdateProcessingStepMessage.class, message -> { + updateProcessingStep(message); + }); } @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private String taskLogContents(SingleTaskLogRequest request) { File taskLogFile = new File(request.getTaskLogInformation().getFullPath()); String stepLogContents = ""; - log.info("Reading task log: " + taskLogFile); + log.info("Reading task log {}", taskLogFile); Charset defaultCharset = null; StringBuilder fileContents = new StringBuilder(1024 * 4); try { fileContents.append(FileUtils.readFileToString(taskLogFile, defaultCharset)); - log.info("Returning task log (" + fileContents.length() + " chars): " + taskLogFile); + log.info("Returning task log {} ({} chars) ", taskLogFile, fileContents.length()); stepLogContents = fileContents.toString(); } catch (IOException e) { throw new UncheckedIOException("Unable to read file " + taskLogFile.toString(), e); @@ -273,31 +285,34 @@ private String taskLogContents(SingleTaskLogRequest request) { } /** - * Kills tasks that run remotely. The {@link AlgorithmManager} for remote jobs is queried for - * the {@link StateFile} instances of all the tasks it's monitoring. If there are any such, it - * uses the PBS qdel command to delete all such tasks from PBS. + * Kills tasks that run remotely. The {@link AlgorithmManager} is queried for remote jobs of all + * the tasks it's monitoring. If there are any such, it uses the PBS qdel command to delete all + * such tasks from PBS. *

* Package scoped for unit testing. */ - void killRemoteTasks(KillTasksRequest message) { - Collection stateFiles = remoteTaskStateFiles(); - if (CollectionUtils.isEmpty(stateFiles)) { + void haltRemoteTasks(HaltTasksRequest message) { + Map> jobIdsByTask = jobIdsByTask(message.getPipelineTasks()); + if (jobIdsByTask.isEmpty()) { return; } - for (StateFile stateFile : stateFiles) { - if (message.getTaskIds().contains(stateFile.getPipelineTaskId()) - && getQueueCommandManager().deleteJobsForStateFile(stateFile) == 0) { - publishKilledTaskMessage(message, stateFile.getPipelineTaskId()); + for (PipelineTask pipelineTask : message.getPipelineTasks()) { + List jobIds = jobIdsByTask.get(pipelineTask); + if (CollectionUtils.isEmpty(jobIds)) { + continue; + } + if (queueCommandManager().deleteJobsByJobId(jobIds) == 0) { + publishTaskHaltedMessage(pipelineTask); } } } - /** Sends killed-task message. */ + /** Sends task-halted message. */ // TODO Inline // This requires that the ZiggyMessenger.publish() call can be verified by Mockito. // The KillTaskMessage.timeSent field makes it difficult. - void publishKilledTaskMessage(KillTasksRequest message, long taskId) { - ZiggyMessenger.publish(new KilledTaskMessage(message, taskId)); + void publishTaskHaltedMessage(PipelineTask pipelineTask) { + ZiggyMessenger.publish(new TaskHaltedMessage(pipelineTask)); } /** @@ -311,15 +326,15 @@ void publishKilledTaskMessage(KillTasksRequest message, long taskId) { *

* Package scoped for unit testing. */ - void handleKilledTaskMessage(KilledTaskMessage message) { - killedTaskIds.add(message.getTaskId()); + void handleTaskHaltedMessage(TaskHaltedMessage message) { + haltedTasks.add(message.getPipelineTask()); // Issue an alert. - alertService().generateAndBroadcastAlert("PI", message.getTaskId(), - AlertService.Severity.ERROR, "Task " + message.getTaskId() + " halted"); + alertService().generateAndBroadcastAlert("PI", message.getPipelineTask(), Severity.ERROR, + "Task " + message.getPipelineTask().getId() + " halted"); // Set the task's error flag. - pipelineTaskOperations().taskErrored(message.getTaskId()); + pipelineTaskDataOperations().taskErrored(message.getPipelineTask()); } /** @@ -328,8 +343,8 @@ void handleKilledTaskMessage(KilledTaskMessage message) { * resubmit, the resubmit counts on the tasks must be reset to zero. */ void restartTasks(RestartTasksRequest message) { - List tasksToResubmit = pipelineTaskOperations() - .prepareTasksForManualResubmit(message.getPipelineTaskIds()); + Collection tasksToResubmit = message.getPipelineTasks(); + pipelineTaskDataOperations().prepareTasksForManualResubmit(tasksToResubmit); pipelineExecutor().restartFailedTasks(tasksToResubmit, message.isDoTransitionOnly(), message.getRunMode()); @@ -338,6 +353,12 @@ void restartTasks(RestartTasksRequest message) { ZiggyMessenger.publish(new InvalidateConsoleModelsMessage()); } + /** Perform pipeline task processing steps serially to avoid race conditions */ + private synchronized void updateProcessingStep(UpdateProcessingStepMessage message) { + pipelineTaskDataOperations().updateProcessingStep(message.getPipelineTask(), + message.getProcessingStep()); + } + public static CommandLine supervisorCommand(WrapperCommand cmd, int workerCount, int workerHeapSize) { Path supervisorPath = DirectoryProperties.ziggyBinDir().resolve(SUPERVISOR_BIN_NAME); @@ -396,28 +417,24 @@ public static Set serializableZiggyEventHandler .collect(Collectors.toSet()); } - public static boolean taskOnKilledTaskList(long taskId) { - return killedTaskIds.contains(taskId); + public static boolean taskOnHaltedTaskList(PipelineTask pipelineTask) { + return haltedTasks.contains(pipelineTask); } - public static void removeTaskFromKilledTaskList(long taskId) { - killedTaskIds.remove(taskId); + public static void removeTaskFromHaltedTaskList(PipelineTask pipelineTask) { + haltedTasks.remove(pipelineTask); } public static WorkerResources defaultResources() { return defaultResources; } - // Package scoped for testing purposes. - Collection remoteTaskStateFiles() { - return AlgorithmMonitor.remoteTaskStateFiles(); + Map> jobIdsByTask(Collection pipelineTasks) { + return algorithmMonitor.jobIdsByTaskId(pipelineTasks); } // Package scoped for testing purposes. - QueueCommandManager getQueueCommandManager() { - if (queueCommandManager == null) { - queueCommandManager = QueueCommandManager.newInstance(); - } + QueueCommandManager queueCommandManager() { return queueCommandManager; } @@ -429,6 +446,10 @@ PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + PipelineInstanceOperations pipelineInstanceOperations() { return pipelineInstanceOperations; } diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskContext.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskContext.java deleted file mode 100644 index 413ac98..0000000 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskContext.java +++ /dev/null @@ -1,110 +0,0 @@ -package gov.nasa.ziggy.supervisor; - -import java.util.Map; - -import gov.nasa.ziggy.metrics.Metric; -import gov.nasa.ziggy.pipeline.definition.PipelineModule; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.uow.UnitOfWork; - -/** - * @author Todd Klaus - */ -public class TaskContext { - public enum TaskState { - IDLE, PROCESSING - } - - private Long pipelineInstanceId; - private Long pipelineTaskId; - private PipelineModule pipelineModule = null; - private TaskState state = TaskState.IDLE; - private String module = "-"; - private String moduleUow = "-"; - private long processingStartTimeMillis = 0; - private long moduleExecTime = 0L; - private Map taskMetrics = null; - - public TaskContext() { - } - - public void setTask(PipelineTask pipelineTask) { - pipelineTaskId = pipelineTask.getId(); - pipelineInstanceId = pipelineTask.getPipelineInstanceId(); - module = pipelineTask.getModuleName(); - UnitOfWork uow = pipelineTask.uowTaskInstance(); - moduleUow = uow.briefState(); - } - - public Long getPipelineInstanceId() { - return pipelineInstanceId; - } - - public void setPipelineInstanceId(Long pipelineInstanceId) { - this.pipelineInstanceId = pipelineInstanceId; - } - - public Long getPipelineTaskId() { - return pipelineTaskId; - } - - public void setPipelineTaskId(Long pipelineTaskId) { - this.pipelineTaskId = pipelineTaskId; - } - - public PipelineModule getPipelineModule() { - return pipelineModule; - } - - public void setPipelineModule(PipelineModule currentPipelineModule) { - pipelineModule = currentPipelineModule; - } - - public TaskState getState() { - return state; - } - - public void setState(TaskState currentState) { - state = currentState; - } - - public String getModule() { - return module; - } - - public void setModule(String currentModule) { - module = currentModule; - } - - public String getModuleUow() { - return moduleUow; - } - - public void setModuleUow(String currentModuleUow) { - moduleUow = currentModuleUow; - } - - public long getProcessingStartTimeMillis() { - return processingStartTimeMillis; - } - - public void setProcessingStartTimeMillis(long currentProcessingStartTimeMillis) { - processingStartTimeMillis = currentProcessingStartTimeMillis; - } - - public long getModuleExecTime() { - return moduleExecTime; - } - - public void setModuleExecTime(long moduleExecTime) { - this.moduleExecTime = moduleExecTime; - } - - public Map getTaskMetrics() { - return taskMetrics; - } - - public void setTaskMetrics(Map taskMetrics) { - this.taskMetrics = taskMetrics; - } -} diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java index 7dab969..d014549 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java @@ -18,7 +18,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.logging.TaskLog; @@ -53,6 +55,7 @@ public class TaskRequestHandler implements Runnable, Requestor { private final long pipelineDefinitionNodeId; private final CountDownLatch taskRequestThreadCountdownLatch; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); private PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); public TaskRequestHandler(int workerId, int workerCount, int heapSizeMb, @@ -79,7 +82,7 @@ public TaskRequestHandler(int workerId, int workerCount, int heapSizeMb, @Override @AcceptableCatchBlock(rationale = Rationale.CLEANUP_BEFORE_EXIT) public void run() { - log.debug("run() - start"); + log.debug("Starting"); try { while (!Thread.currentThread().isInterrupted()) { @@ -96,9 +99,9 @@ public void run() { // number of workers, all the TaskRequestHandlers should shut down when they see // that this has happened. if (taskRequest.getPipelineDefinitionNodeId() != pipelineDefinitionNodeId) { - log.info("Transitioning to pipeline definition node " - + taskRequest.getPipelineDefinitionNodeId() - + " so shutting down task request handler"); + log.info( + "Transitioning to pipeline definition node {} so shutting down task request handler", + taskRequest.getPipelineDefinitionNodeId()); // Also, in this case, the task has to be put back onto the queue. taskRequestQueue.put(taskRequest); @@ -134,7 +137,7 @@ private void processTaskRequest(TaskRequest taskRequest) { // the instance node means that the tasks failed and the pipeline transitioned to // doing nothing, and we are now rerunning the tasks and want to be able to // transition to the next node. - log.info("start: taskRequest={}", taskRequest); + log.info("Start processing taskRequest={}", taskRequest); markInstanceNodeNotTransitioned(taskRequest.getInstanceNodeId()); ExternalProcess externalProcess = ExternalProcess @@ -150,25 +153,21 @@ private void processTaskRequest(TaskRequest taskRequest) { // If the external process returned a nonzero status, we need to ensure that // the task is correctly marked as errored and that the instance is properly // updated. + PipelineTask pipelineTask = taskRequest.getPipelineTask(); if (status != 0) { - PipelineTask pipelineTask = pipelineTaskOperations() - .pipelineTask(taskRequest.getTaskId()); log.error("Marking task {} failed because PipelineWorker return value is {}", - taskRequest.getTaskId(), status); + pipelineTask, status); AlertService alertService = AlertService.getInstance(); - alertService.generateAndBroadcastAlert("PI", taskRequest.getTaskId(), - AlertService.Severity.ERROR, + alertService.generateAndBroadcastAlert("PI", pipelineTask, Severity.ERROR, "Task marked as errored due to WorkerProcess return value " + status); - if (!pipelineTask.isError()) { - pipelineTaskOperations().taskErrored(pipelineTask); - } + pipelineTaskDataOperations().taskErrored(pipelineTask); } // Finalize the task's memdrone activities. - PipelineTask pipelineTask = pipelineTaskOperations().pipelineTask(taskRequest.getTaskId()); - log.info("task {}: retried={}, errored={}", pipelineTask.getId(), pipelineTask.isRetry(), - pipelineTask.isError()); + boolean taskErrored = pipelineTaskDataOperations().hasErrored(pipelineTask); + log.info("task={}, retried={}, errored={}", pipelineTask, + pipelineTaskDataOperations().retrying(pipelineTask), taskErrored); Memdrone memdrone = new Memdrone(pipelineTask.getModuleName(), pipelineTask.getPipelineInstanceId()); @@ -178,7 +177,7 @@ private void processTaskRequest(TaskRequest taskRequest) { } transitionToNextInstanceNode(taskRequest.getInstanceNodeId()); - log.info("end: taskRequest={}", taskRequest); + log.info("Finished processing taskRequest={}", taskRequest); } /** @@ -210,7 +209,7 @@ private static synchronized void transitionToNextInstanceNode(long instanceNodeI log.info("Node {} execution complete", pipelineInstanceNode.getModuleName()); new PipelineInstanceNodeOperations().markInstanceNodeTransitionComplete(instanceNodeId); - log.info("Task counts: {}", taskCounts); + log.info("{}", taskCounts); if (taskCounts.isPipelineTasksComplete()) { pipelineExecutor.transitionToNextInstanceNode(pipelineInstanceNode); } else { @@ -253,16 +252,16 @@ private CommandLine commandLine(TaskRequest taskRequest) { int processHeapSizeMb = Math.round((float) heapSizeMb / workerCount); commandLine.addArgument("-Xmx" + Integer.toString(processHeapSizeMb) + "M"); - commandLine.addArgument(TaskLog.ziggyLogFileSystemProperty( - pipelineTaskOperations().pipelineTask(taskRequest.getTaskId()))); + PipelineTask pipelineTask = taskRequest.getPipelineTask(); + commandLine.addArgument(TaskLog.ziggyLogFileSystemProperty(pipelineTask)); // Now for the worker process fully qualified class name commandLine.addArgument("--class=" + PipelineWorker.class.getName()); // Now for the worker arguments commandLine.addArgument(Integer.toString(workerId)); - commandLine.addArgument(Long.toString(taskRequest.getTaskId())); + commandLine.addArgument(pipelineTask.toString()); commandLine.addArgument(taskRequest.getRunMode().name()); - log.debug("Worker command line: " + commandLine.toString()); + log.debug("commandLine={}", commandLine.toString()); return commandLine; } @@ -276,6 +275,10 @@ PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + PipelineInstanceNodeOperations pipelineInstanceOperations() { return pipelineInstanceNodeOperations; } diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java index 4c6763f..2f8bbc6 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java @@ -13,10 +13,12 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.services.messages.KilledTaskMessage; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; +import gov.nasa.ziggy.services.messages.TaskHaltedMessage; import gov.nasa.ziggy.services.messages.TaskRequest; import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; @@ -38,6 +40,7 @@ * one instance should be constructed and stored in {@link PipelineSupervisor}. * * @author PT + * @author Bill Wohler */ public class TaskRequestHandlerLifecycleManager extends Thread { @@ -64,7 +67,8 @@ public class TaskRequestHandlerLifecycleManager extends Thread { private List> taskRequestHandlers = new ArrayList<>(); private final boolean storeTaskRequestHandlers; - private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); private TaskRequestHandlerLifecycleManager() { this(false); @@ -75,12 +79,12 @@ protected TaskRequestHandlerLifecycleManager(boolean storeTaskRequestHandlers) { // Subscribe to task request messages. ZiggyMessenger.subscribe(TaskRequest.class, message -> { - taskRequestQueue.put(message); + handleTaskRequestAction(message); }); // Subscribe to kill tasks messages. - ZiggyMessenger.subscribe(KillTasksRequest.class, message -> { - killQueuedTasksAction(message); + ZiggyMessenger.subscribe(HaltTasksRequest.class, message -> { + haltQueuedTasksAction(message); }); // Subscribe to requests for the current worker resources. This allows the @@ -104,53 +108,66 @@ public static synchronized void initializeInstance() { instance.start(); } + // Package scoped to facilitate testing. + void handleTaskRequestAction(TaskRequest taskRequest) { + if (pipelineTaskDataOperations().haltRequested(taskRequest.getPipelineTask())) { + publishTaskHaltedMessage(taskRequest.getPipelineTask()); + return; + } + taskRequestQueue.put(taskRequest); + } + /** * Performs the deletion of queued tasks. This consists of removing the task requests from the - * task request queue and setting the error flag of the removed tasks. The IDs of all tasks - * listed in the {@link KillTasksRequest} are added to the supervisor's list of killed task IDs. - * This avoids the need to capture the IDs from the worker, the supervisor, and the batch - * queues. + * task request queue and setting the error flag of the removed tasks. The tasks listed in the + * {@link HaltTasksRequest} are added to the supervisor's list of halted tasks. This avoids the + * need to capture the tasks from the worker, the supervisor, and the batch queues. *

* This method is package scoped to facilitate testing. */ - void killQueuedTasksAction(KillTasksRequest request) { - List taskIds = request.getTaskIds(); + void haltQueuedTasksAction(HaltTasksRequest request) { + List pipelineTasks = request.getPipelineTasks(); // Locate the tasks that are queued and which are in the kill request. - log.info("Halting queued tasks: {}", taskIds); - List tasksForDeletion = new ArrayList<>(); - Set taskIdsForDeletion = new HashSet<>(); + log.info("Halting queued tasks: {}", pipelineTasks); + List taskRequestsForDeletion = new ArrayList<>(); + Set tasksForDeletion = new HashSet<>(); for (TaskRequest taskRequest : taskRequestQueue) { - if (taskIds.contains(taskRequest.getTaskId())) { - tasksForDeletion.add(taskRequest); - taskIdsForDeletion.add(taskRequest.getTaskId()); + PipelineTask pipelineTask = taskRequest.getPipelineTask(); + if (pipelineTasks.contains(pipelineTask)) { + taskRequestsForDeletion.add(taskRequest); + tasksForDeletion.add(pipelineTask); } } // Take the selected tasks out of the queue. - taskRequestQueue.removeAll(tasksForDeletion); + taskRequestQueue.removeAll(taskRequestsForDeletion); // Removing a task from the queue is easy, so we can just assume this was successful // and publish the success messages. - for (TaskRequest taskRequest : tasksForDeletion) { - publishKilledTaskMessage(request, taskRequest.getTaskId()); + for (TaskRequest taskRequest : taskRequestsForDeletion) { + publishTaskHaltedMessage(taskRequest.getPipelineTask()); } } /** - * Publishes the {@link KilledTaskMessage}. + * Publishes the {@link TaskHaltedMessage}. */ // TODO Inline // This requires that the ZiggyMessenger.publish() call can be verified by Mockito. // The KillTaskMessage.timeSent field makes it difficult. - void publishKilledTaskMessage(KillTasksRequest request, long taskId) { - ZiggyMessenger.publish(new KilledTaskMessage(request, taskId)); + void publishTaskHaltedMessage(PipelineTask pipelineTask) { + ZiggyMessenger.publish(new TaskHaltedMessage(pipelineTask)); } protected PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + protected PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + protected AlertService alertService() { return AlertService.getInstance(); } @@ -207,10 +224,10 @@ private void startLifecycle() { workerResources.humanReadableHeapSize().toString()); taskRequestThreadPool = Executors.newFixedThreadPool(workerCount); List handlers = new ArrayList<>(); - for (int i = 0; i < workerCount; i++) { - log.info("Starting worker # {} of {}", i + 1, workerCount); - TaskRequestHandler handler = new TaskRequestHandler(i + 1, workerCount, - heapSizeMb, taskRequestQueue, pipelineDefinitionNodeId, + for (int i = 1; i <= workerCount; i++) { + log.info("Starting worker {} of {}", i, workerCount); + TaskRequestHandler handler = new TaskRequestHandler(i, workerCount, heapSizeMb, + taskRequestQueue, pipelineDefinitionNodeId, taskRequestThreadCountdownLatch); taskRequestThreadPool.submit(handler); handlers.add(handler); @@ -247,7 +264,7 @@ public void shutdown() { */ WorkerResources workerResources(TaskRequest taskRequest) { WorkerResources databaseResources = pipelineTaskOperations() - .workerResourcesForTask(taskRequest.getTaskId()); + .workerResourcesForTask(taskRequest.getPipelineTask()); Integer compositeWorkerCount = databaseResources.getMaxWorkerCount() != null ? databaseResources.getMaxWorkerCount() : PipelineSupervisor.defaultResources().getMaxWorkerCount(); diff --git a/src/main/java/gov/nasa/ziggy/ui/ClusterController.java b/src/main/java/gov/nasa/ziggy/ui/ClusterController.java index 2c7683d..f3f6f09 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ClusterController.java +++ b/src/main/java/gov/nasa/ziggy/ui/ClusterController.java @@ -72,6 +72,7 @@ import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; import gov.nasa.ziggy.services.process.ExternalProcess; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.WrapperUtils.WrapperCommand; import gov.nasa.ziggy.util.io.ZiggyFileUtils; @@ -197,8 +198,7 @@ private int cpuCount() { public static void main(String[] args) { - log.info("Ziggy cluster (version {})", - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + log.info("Ziggy cluster (version {})", BuildInfo.ziggyVersion()); // Define all the command options. Options options = new Options() @@ -297,8 +297,7 @@ public static void main(String[] args) { } if (commands.contains(VERSION_COMMAND)) { - System.out.println( - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + System.out.println(BuildInfo.ziggyVersion()); } } @@ -373,7 +372,7 @@ private void initializeCluster(boolean force) throws Exception { name) -> (name.startsWith(PARAM_LIBRARY_PREFIX) && name.endsWith(XML_SUFFIX))); Arrays.sort(parameterFiles, Comparator.comparing(File::getName)); - log.info("Importing datastore configuration from directory " + pipelineDefsDir.toString()); + log.info("Importing datastore configuration from directory {}", pipelineDefsDir.toString()); File[] dataTypeFiles = pipelineDefsDir.toFile() .listFiles((FilenameFilter) (dir, name) -> (name.startsWith(TYPE_FILE_PREFIX) && name.endsWith(XML_SUFFIX))); diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java index 4dd5c98..8507caa 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java @@ -63,16 +63,17 @@ import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.services.alert.AlertLog; import gov.nasa.ziggy.services.alert.AlertLogOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.messages.StartPipelineRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; @@ -80,6 +81,7 @@ import gov.nasa.ziggy.ui.util.TaskRestarter; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.ZiggyShutdownHook; import gov.nasa.ziggy.util.ZiggyStringUtils; import gov.nasa.ziggy.util.dispmod.AlertLogDisplayModel; @@ -162,6 +164,8 @@ version Display the version (as a Git tag) private final PipelineDefinitionOperations pipelineDefinitionOperations = new PipelineDefinitionOperations(); private final PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); @AcceptableCatchBlock(rationale = Rationale.USAGE) @AcceptableCatchBlock(rationale = Rationale.SYSTEM_EXIT) @@ -340,8 +344,7 @@ private void runCommand(Command command, CommandLine cmdLine, List comma case LOG -> log(); case RESTART -> restart(cmdLine); case START -> start(commands); - case VERSION -> System.out.println(ZiggyConfiguration.getInstance() - .getString(PropertyName.ZIGGY_VERSION.property())); + case VERSION -> System.out.println(BuildInfo.ziggyVersion()); } } catch (Throwable e) { exception = e; @@ -454,7 +457,7 @@ private int showPipelineNode(PipelineDefinitionNode node, int index) { private void display(CommandLine cmdLine) { PipelineInstance instance = null; - PipelineTask task = null; + PipelineTaskDisplayData task = null; if (cmdLine.hasOption(INSTANCE_OPTION)) { instance = pipelineInstance(cmdLine.getOptionValue(INSTANCE_OPTION)); } else if (cmdLine.hasOption(TASK_OPTION)) { @@ -503,12 +506,13 @@ private PipelineInstance pipelineInstance(String instanceOption) { return instance; } - private PipelineTask pipelineTask(String taskOption) { + private PipelineTaskDisplayData pipelineTask(String taskOption) { return pipelineTask(parseId(taskOption)); } - private PipelineTask pipelineTask(long id) { - PipelineTask task = pipelineTaskOperations().pipelineTask(id); + private PipelineTaskDisplayData pipelineTask(long id) { + PipelineTaskDisplayData task = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(pipelineTaskOperations().pipelineTask(id)); if (task == null) { throw new PipelineException("No task found with ID " + id); } @@ -535,16 +539,17 @@ private DisplayType parseDisplayType(CommandLine cmdLine) { } private boolean displayAlert(PipelineInstance instance) { - List alerts = alertLogOperations().alertLogs(instance.getId()); + List alerts = alertLogOperations().alertLogs(instance); AlertLogDisplayModel alertLogDisplayModel = new AlertLogDisplayModel(alerts); alertLogDisplayModel.print(System.out, "Alerts"); return true; } private boolean displayErrors(PipelineInstance instance) { - List tasks = pipelineTaskOperations().erroredPipelineTasks(instance); + List tasks = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(pipelineTaskDataOperations().erroredPipelineTasks(instance)); - for (PipelineTask task : tasks) { + for (PipelineTaskDisplayData task : tasks) { TasksDisplayModel tasksDisplayModel = new TasksDisplayModel(task); tasksDisplayModel.print(System.out, "Task Summary"); } @@ -552,7 +557,8 @@ private boolean displayErrors(PipelineInstance instance) { } private boolean displayStatistics(PipelineInstance instance) { - List tasks = pipelineTaskOperations().pipelineTasks(instance); + List tasks = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(instance); List orderedModuleNames = displayTaskSummary(instance, false).getModuleNames(); PipelineStatsDisplayModel pipelineStatsDisplayModel = new PipelineStatsDisplayModel(tasks, @@ -583,7 +589,8 @@ private long parseId(String id) { } private TaskCounts displayTaskSummary(PipelineInstance instance, boolean full) { - List tasks = pipelineTaskOperations().pipelineTasks(instance); + List tasks = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(instance); TaskSummaryDisplayModel taskSummaryDisplayModel = new TaskSummaryDisplayModel( new TaskCounts(tasks)); taskSummaryDisplayModel.print(System.out, "Instance Task Summary"); @@ -614,15 +621,17 @@ private void halt(CommandLine cmdLine) { if (cmdLine.hasOption(INSTANCE_OPTION)) { pipelineInstance = pipelineInstance(cmdLine.getOptionValue(INSTANCE_OPTION)); } else { - PipelineTask pipelineTask = pipelineTask(taskIds.get(0)); - pipelineInstance = pipelineTaskOperations().pipelineInstance(pipelineTask); + pipelineInstance = pipelineTaskOperations().pipelineInstance(taskIds.get(0)); } System.out.println("Halting task(s) " + taskIds.stream().map(Object::toString).collect(Collectors.joining(", ")) + " in instance " + pipelineInstance.getId()); CountDownLatch messageSentLatch = startZiggyClient(); - new TaskHalter().haltTasks(pipelineInstance, taskIds); + new TaskHalter().haltTasks(pipelineInstance, + taskIds.stream() + .map(id -> pipelineTaskOperations().pipelineTask(id)) + .collect(Collectors.toList())); messageSentLatch.countDown(); } @@ -666,8 +675,7 @@ private void restart(CommandLine cmdLine) { if (cmdLine.hasOption(INSTANCE_OPTION)) { pipelineInstance = pipelineInstance(cmdLine.getOptionValue(INSTANCE_OPTION)); } else { - PipelineTask pipelineTask = pipelineTask(taskIds.get(0)); - pipelineInstance = pipelineTaskOperations().pipelineInstance(pipelineTask); + pipelineInstance = pipelineTaskOperations().pipelineInstance(taskIds.get(0)); } // Get the run mode, using Restart from beginning if none provided. @@ -678,7 +686,10 @@ private void restart(CommandLine cmdLine) { // Create two latches to wait for the restart messages. CountDownLatch clientStillNeededLatch = startZiggyClient(2); - new TaskRestarter().restartTasks(pipelineInstance, taskIds, runMode, + List pipelineTasks = taskIds.stream() + .map(id -> pipelineTaskOperations().pipelineTask(id)) + .collect(Collectors.toList()); + new TaskRestarter().restartTasks(pipelineInstance, pipelineTasks, runMode, clientStillNeededLatch); } @@ -763,4 +774,12 @@ private PipelineInstanceOperations pipelineInstanceOperations() { private PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + private PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java index 627c347..1c4d1e0 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java @@ -34,7 +34,6 @@ import gov.nasa.ziggy.ui.parameters.ViewEditParameterSetsPanel; import gov.nasa.ziggy.ui.pipeline.ViewEditPipelinesPanel; import gov.nasa.ziggy.ui.status.StatusPanel; -import gov.nasa.ziggy.ui.util.ViewEditKeyValuePairPanel; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; /** @@ -65,8 +64,7 @@ private enum ContentItem { DATA_RECEIPT("Data Receipt", DataReceiptPanel::new), EVENT_HANDLERS("Event Definitions", ZiggyEventHandlerPanel::new), MODULE_LIBRARY("Module Library", ViewEditModuleLibraryPanel::new), - METRILYZER("Metrilyzer", false, true, MetrilyzerPanel::new), - GENERAL("General", false, true, ViewEditKeyValuePairPanel::new); + METRILYZER("Metrilyzer", false, true, MetrilyzerPanel::new); private String label; private boolean visible; diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java index 7f9dc4b..00424b3 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java @@ -52,6 +52,7 @@ import gov.nasa.ziggy.ui.util.MessageUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.util.Requestor; +import gov.nasa.ziggy.util.BuildInfo; import gov.nasa.ziggy.util.ZiggyShutdownHook; import gov.nasa.ziggy.worker.WorkerResources; @@ -118,8 +119,7 @@ private ZiggyGuiConsole() { public static void launch() { try { - log.info("Starting Ziggy Console ({})", - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + log.info("Starting Ziggy Console ({})", BuildInfo.ziggyVersion()); ZiggyConfiguration.logJvmProperties(); @@ -189,8 +189,7 @@ private String content() { .append( "Ziggy, a portable, scalable infrastructure for science data processing pipelines") .append("

Software version: ") - .append(ZiggyConfiguration.getInstance() - .getString(PropertyName.ZIGGY_VERSION.property())) + .append(BuildInfo.ziggyVersion()) .append("
URL: ") .append("https://github.com/nasa/ziggy") .toString(); @@ -311,7 +310,7 @@ private static Image getImage(String imageLocation) { try { return getImage(new File(imageLocation).toURI().toURL()); } catch (MalformedURLException e) { - log.warn("Bad URL formed from " + imageLocation, e); + log.warn("Bad URL formed from {}", imageLocation, e); return null; } } @@ -321,7 +320,7 @@ private static Image getImage(URL url) { try { image = ImageIO.read(url); } catch (IOException e) { - log.warn("Unable to load image from file " + url.toString(), e); + log.warn("Unable to load image from file {}", url.toString(), e); } return image; } diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConstants.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConstants.java index d308621..5b5776e 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConstants.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConstants.java @@ -60,9 +60,7 @@ public class ZiggyGuiConstants { public static final String CANCEL = "Cancel"; public static final String CLOSE = "Close"; public static final String COLLAPSE_ALL = "Collapse all"; - public static final String COPY = "Copy"; public static final String CREATE = "Create"; - public static final String DELETE = "Delete"; public static final String DELETE_SYMBOL = "-"; public static final String DIALOG = "..."; public static final String EDIT = "Edit"; @@ -72,11 +70,9 @@ public class ZiggyGuiConstants { public static final String FILE = "File"; public static final String HELP = "Help"; public static final String IMPORT = "Import"; - public static final String NEW = "+ New"; public static final String NEW_SYMBOL = "+"; public static final String OK = "OK"; public static final String REFRESH = "Refresh"; - public static final String RENAME = "Rename"; public static final String REPORT = "Report"; public static final String RESTART = "Restart"; public static final String RESTORE_DEFAULTS = "Restore defaults"; diff --git a/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java b/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java index 55457d4..83b8978 100644 --- a/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java @@ -3,9 +3,7 @@ import static com.google.common.base.Preconditions.checkArgument; import java.util.ArrayList; -import java.util.HashSet; import java.util.List; -import java.util.Set; import java.util.concurrent.ExecutionException; import javax.swing.SwingUtilities; @@ -49,11 +47,6 @@ protected void refresh() { } } - @Override - protected void create() { - throw new UnsupportedOperationException("Create not supported"); - } - @Override protected void edit(int row) { DatastoreRegexp regexp = ziggyTable.getContentAtViewRow(row); @@ -66,16 +59,6 @@ protected void edit(int row) { } } - @Override - protected void delete(int row) { - throw new UnsupportedOperationException("Delete not supported"); - } - - @Override - protected Set optionalViewEditFunctions() { - return new HashSet<>(); - } - public static class RegexpTableModel extends AbstractZiggyTableModel implements DatabaseModel { @@ -87,6 +70,26 @@ public static class RegexpTableModel extends AbstractZiggyTableModel, Void>() { + @Override + protected List doInBackground() throws Exception { + return datastoreOperations().datastoreRegexps(); + } + + @Override + protected void done() { + try { + datastoreRegexps = get(); + fireTableDataChanged(); + } catch (InterruptedException | ExecutionException e) { + log.error("Could not retrieve datastore regexps", e); + } + } + }.execute(); + } + @Override public int getRowCount() { return datastoreRegexps.size(); @@ -121,26 +124,6 @@ public Object getValueAt(int rowIndex, int columnIndex) { }; } - @Override - public void loadFromDatabase() { - new SwingWorker, Void>() { - @Override - protected List doInBackground() throws Exception { - return datastoreOperations().datastoreRegexps(); - } - - @Override - protected void done() { - try { - datastoreRegexps = get(); - fireTableDataChanged(); - } catch (InterruptedException | ExecutionException e) { - log.error("Could not retrieve datastore regexps", e); - } - } - }.execute(); - } - @Override public DatastoreRegexp getContentAtRow(int row) { return datastoreRegexps.get(row); diff --git a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java index a39cb4e..cc71e28 100644 --- a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java @@ -141,6 +141,13 @@ public DataReceiptInstanceTableModel(DataReceiptInstance dataReceiptInstance) { loadFromDatabase(); } + @Override + public void loadFromDatabase() { + dataReceiptFiles = dataReceiptOperations() + .dataReceiptFilesForInstance(dataReceiptInstance.getInstanceId()); + fireTableDataChanged(); + } + @Override public int getRowCount() { return dataReceiptFiles.size(); @@ -168,13 +175,6 @@ public Object getValueAt(int rowIndex, int columnIndex) { }; } - @Override - public void loadFromDatabase() { - dataReceiptFiles = dataReceiptOperations() - .dataReceiptFilesForInstance(dataReceiptInstance.getInstanceId()); - fireTableDataChanged(); - } - @Override public DataReceiptFile getContentAtRow(int row) { return dataReceiptFiles.get(row); diff --git a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptPanel.java b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptPanel.java index 0b9a412..d8cf3e0 100644 --- a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptPanel.java @@ -116,6 +116,26 @@ private static class DataReceiptTableModel extends AbstractZiggyTableModel, Void>() { + @Override + protected List doInBackground() throws Exception { + return dataReceiptOperations().dataReceiptInstances(); + } + + @Override + protected void done() { + try { + dataReceiptInstances = get(); + fireTableDataChanged(); + } catch (InterruptedException | ExecutionException e) { + log.error("Could not load data receipt objects", e); + } + } + }.execute(); + } + @Override public int getRowCount() { return dataReceiptInstances.size(); @@ -144,26 +164,6 @@ public Object getValueAt(int rowIndex, int columnIndex) { }; } - @Override - public void loadFromDatabase() { - new SwingWorker, Void>() { - @Override - protected List doInBackground() throws Exception { - return dataReceiptOperations().dataReceiptInstances(); - } - - @Override - protected void done() { - try { - dataReceiptInstances = get(); - fireTableDataChanged(); - } catch (InterruptedException | ExecutionException e) { - log.error("Could not load data receipt objects", e); - } - } - }.execute(); - } - @Override public DataReceiptInstance getContentAtRow(int row) { return dataReceiptInstances.get(row); diff --git a/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java b/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java index 234a03f..02416a6 100644 --- a/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java @@ -129,6 +129,16 @@ private static class EventHandlerTableModel extends AbstractTableModel private final UUID uuid = UUID.randomUUID(); private List eventHandlers = new ArrayList<>(); + /** + * Updates the models event handlers. + */ + private void setEventHandlers(Collection handlerInfo) { + log.debug("Updating model"); + eventHandlers.clear(); + eventHandlers.addAll(handlerInfo); + fireTableDataChanged(); + } + public String getName(int rowIndex) { return eventHandlers.get(rowIndex).getName(); } @@ -177,16 +187,6 @@ public Class tableModelContentClass() { public UUID requestorIdentifier() { return uuid; } - - /** - * Updates the models event handlers. - */ - private void setEventHandlers(Collection handlerInfo) { - log.debug("Updating model"); - eventHandlers.clear(); - eventHandlers.addAll(handlerInfo); - fireTableDataChanged(); - } } /** diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/DisplayTasksForInstanceMessage.java b/src/main/java/gov/nasa/ziggy/ui/instances/DisplayTasksForInstanceMessage.java deleted file mode 100644 index eda4ac6..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/instances/DisplayTasksForInstanceMessage.java +++ /dev/null @@ -1,30 +0,0 @@ -package gov.nasa.ziggy.ui.instances; - -import gov.nasa.ziggy.services.messages.PipelineMessage; - -/** - * Message sent from the {@link InstancesPanel} to the {@link TasksPanel} notifying the latter that - * it may need to update its displays because of a change of the selected instance. - * - * @author PT - */ -public class DisplayTasksForInstanceMessage extends PipelineMessage { - - private static final long serialVersionUID = 20230712L; - - private final boolean reselect; - private final Long pipelineInstanceId; - - public DisplayTasksForInstanceMessage(boolean reselect, Long pipelineInstanceId) { - this.reselect = reselect; - this.pipelineInstanceId = pipelineInstanceId; - } - - public boolean isReselect() { - return reselect; - } - - public long getPipelineInstanceId() { - return pipelineInstanceId; - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java index 5dd6bf4..af465a6 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java @@ -23,7 +23,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.ui.ZiggyGuiConstants; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; import gov.nasa.ziggy.ui.util.table.ZiggyTable; @@ -40,7 +42,8 @@ public class InstanceCostEstimateDialog extends JDialog { private static final long serialVersionUID = 20240614L; - private final PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); public InstanceCostEstimateDialog(Window owner, PipelineInstance pipelineInstance) { super(owner, DEFAULT_MODALITY_TYPE); @@ -71,17 +74,18 @@ private JPanel createDataPanel(PipelineInstance pipelineInstance) { JLabel state = boldLabel("State:"); JLabel stateText = new JLabel(pipelineInstance.getState().toString()); - List pipelineTasksInInstance = pipelineInstanceOperations() - .updateJobs(pipelineInstance); + pipelineTaskDataOperations().updateJobs(pipelineInstance); + List pipelineTaskDataInInstance = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(pipelineInstance); JLabel cost = boldLabel("Cost estimate:"); - JLabel costText = new JLabel(instanceCost(pipelineTasksInInstance)); + JLabel costText = new JLabel(instanceCost(pipelineTaskDataInInstance)); JLabel tasks = boldLabel("Pipeline tasks", LabelType.HEADING); tasks.setToolTipText("Displays estimated cost of tasks in the selected pipeline instance."); ZiggyTable ziggyTable = new ZiggyTable<>( - new TaskCostEstimateTableModel(pipelineTasksInInstance)); + new TaskCostEstimateTableModel(pipelineTaskDataInInstance)); ETable taskTable = ziggyTable.getTable(); taskTable.setCellSelectionEnabled(false); @@ -129,9 +133,9 @@ private void close(ActionEvent evt) { setVisible(false); } - static String instanceCost(List pipelineTasksInInstance) { + static String instanceCost(List pipelineTasksInInstance) { double totalCost = 0; - for (PipelineTask task : pipelineTasksInInstance) { + for (PipelineTaskDisplayData task : pipelineTasksInInstance) { totalCost += task.costEstimate(); } return formatCost(totalCost); @@ -152,8 +156,12 @@ private static String formatCost(double cost) { return new DecimalFormat(format).format(cost); } - private PipelineInstanceOperations pipelineInstanceOperations() { - return pipelineInstanceOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; } private static class TaskCostEstimateTableModel extends AbstractTableModel @@ -164,9 +172,9 @@ private static class TaskCostEstimateTableModel extends AbstractTableModel private static final String[] COLUMN_NAMES = { "ID", "Module", "UOW", "Status", "Cost estimate" }; - private final List pipelineTasks; + private final List pipelineTasks; - public TaskCostEstimateTableModel(List pipelineTasks) { + public TaskCostEstimateTableModel(List pipelineTasks) { this.pipelineTasks = pipelineTasks; } @@ -187,11 +195,11 @@ public String getColumnName(int columnIndex) { @Override public Object getValueAt(int rowIndex, int columnIndex) { - PipelineTask task = pipelineTasks.get(rowIndex); + PipelineTaskDisplayData task = pipelineTasks.get(rowIndex); return switch (columnIndex) { - case 0 -> task.getId(); + case 0 -> task.getPipelineTaskId(); case 1 -> task.getModuleName(); - case 2 -> task.uowTaskInstance().briefState(); + case 2 -> task.getBriefState(); case 3 -> task.getDisplayProcessingStep(); case 4 -> formatCost(task.costEstimate()); default -> throw new IllegalArgumentException( diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceDetailsDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceDetailsDialog.java index 802541c..df3e8a3 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceDetailsDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceDetailsDialog.java @@ -9,7 +9,6 @@ import java.awt.BorderLayout; import java.awt.Window; import java.awt.event.ActionEvent; -import java.util.Date; import java.util.HashMap; import java.util.LinkedList; import java.util.List; @@ -32,6 +31,7 @@ import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.ui.ZiggyGuiConstants; import gov.nasa.ziggy.ui.util.MessageUtils; import gov.nasa.ziggy.ui.util.TextualReportDialog; @@ -84,15 +84,10 @@ private JPanel createDataPanel() { nameText = new JLabel(pipelineInstance.getName()); JLabel start = boldLabel("Start:"); - JLabel startText = new JLabel(pipelineInstance.getStartProcessingTime().toString()); - - JLabel end = boldLabel("End:"); - Date endProcessingTime = pipelineInstance.getEndProcessingTime(); - JLabel endText = new JLabel( - endProcessingTime.getTime() == 0 ? "-" : endProcessingTime.toString()); + JLabel startText = new JLabel(pipelineInstance.getCreated().toString()); JLabel total = boldLabel("Total:"); - JLabel totalText = new JLabel(pipelineInstance.elapsedTime()); + JLabel totalText = new JLabel(pipelineInstance.getExecutionClock().toString()); JLabel pipelineParametersGroup = boldLabel("Pipeline parameter sets", LabelType.HEADING1); ParameterSetViewPanel pipelineParameterSetsPanel = new ParameterSetViewPanel( @@ -116,14 +111,12 @@ private JPanel createDataPanel() { .addComponent(id) .addComponent(name) .addComponent(start) - .addComponent(end) .addComponent(total)) .addPreferredGap(ComponentPlacement.RELATED) .addGroup(dataPanelLayout.createParallelGroup() .addComponent(idText) .addComponent(nameText) .addComponent(startText) - .addComponent(endText) .addComponent(totalText))) .addComponent(pipelineParametersGroup) .addComponent(pipelineParameterSetsPanel) @@ -142,8 +135,6 @@ private JPanel createDataPanel() { .addGroup( dataPanelLayout.createParallelGroup().addComponent(start).addComponent(startText)) .addPreferredGap(ComponentPlacement.RELATED) - .addGroup(dataPanelLayout.createParallelGroup().addComponent(end).addComponent(endText)) - .addPreferredGap(ComponentPlacement.RELATED) .addGroup( dataPanelLayout.createParallelGroup().addComponent(total).addComponent(totalText)) .addGap(ZiggyGuiConstants.GROUP_GAP) @@ -229,6 +220,7 @@ private static class InstanceModulesTableModel private Map nodeTaskCounts = new HashMap<>(); private final PipelineInstanceNodeOperations pipelineInstanceNodeOperations = new PipelineInstanceNodeOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); public InstanceModulesTableModel(PipelineInstance instance) { if (instance == null) { @@ -238,7 +230,7 @@ public InstanceModulesTableModel(PipelineInstance instance) { pipelineInstanceNodes = pipelineInstanceNodeOperations() .pipelineInstanceNodes(instance); for (PipelineInstanceNode node : pipelineInstanceNodes) { - nodeTaskCounts.put(node, pipelineInstanceNodeOperations().taskCounts(node)); + nodeTaskCounts.put(node, pipelineTaskDisplayDataOperations().taskCounts(node)); } } @@ -285,5 +277,9 @@ public PipelineInstanceNode getContentAtRow(int row) { private PipelineInstanceNodeOperations pipelineInstanceNodeOperations() { return pipelineInstanceNodeOperations; } + + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java index ca0ab6b..46f2cd0 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java @@ -18,8 +18,8 @@ import javax.swing.table.AbstractTableModel; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; import gov.nasa.ziggy.ui.util.table.ZiggyTable; @@ -40,10 +40,10 @@ public class InstanceStatsDialog extends javax.swing.JDialog { private final PipelineInstance pipelineInstance; private TaskMetricsTableModel processingBreakdownTableModel; private PipelineStatsTableModel processingTimeTableModel; - private List tasks; + private List tasks; private ArrayList orderedModuleNames; - private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); public InstanceStatsDialog(Window owner, PipelineInstance instance) { super(owner, DEFAULT_MODALITY_TYPE); @@ -57,10 +57,10 @@ public InstanceStatsDialog(Window owner, PipelineInstance instance) { } private void loadFromDatabase() { - tasks = pipelineTaskOperations().pipelineTasks(pipelineInstance, true); + tasks = pipelineTaskDisplayDataOperations().pipelineTaskDisplayData(pipelineInstance); orderedModuleNames = new ArrayList<>(); - for (PipelineTask task : tasks) { + for (PipelineTaskDisplayData task : tasks) { String moduleName = task.getModuleName(); if (!orderedModuleNames.contains(moduleName)) { orderedModuleNames.add(moduleName); @@ -138,8 +138,8 @@ private void close(ActionEvent evt) { setVisible(false); } - private PipelineTaskOperations pipelineTaskOperations() { - return pipelineTaskOperations; + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; } private static class PipelineStatsTableModel extends AbstractTableModel @@ -147,11 +147,12 @@ private static class PipelineStatsTableModel extends AbstractTableModel private PipelineStatsDisplayModel pipelineStatsDisplayModel; - public PipelineStatsTableModel(List tasks, List orderedModuleNames) { + public PipelineStatsTableModel(List tasks, + List orderedModuleNames) { update(tasks, orderedModuleNames); } - public void update(List tasks, List orderedModuleNames) { + public void update(List tasks, List orderedModuleNames) { pipelineStatsDisplayModel = new PipelineStatsDisplayModel(tasks, orderedModuleNames); fireTableDataChanged(); } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java index 5137f0d..5065249 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java @@ -13,6 +13,7 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.TreeMap; import java.util.concurrent.ExecutionException; @@ -46,6 +47,7 @@ import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.models.DatabaseModel; +import gov.nasa.ziggy.ui.util.table.TableUpdater; import gov.nasa.ziggy.ui.util.table.ZiggyTable; /** @@ -57,6 +59,9 @@ public class InstancesTable extends JPanel { private static final long serialVersionUID = 20240614L; + @SuppressWarnings("unused") + private static final Logger log = LoggerFactory.getLogger(InstancesTable.class); + private Component parent; private ZiggyTable instancesTable; private InstancesTableModel instancesTableModel; @@ -91,18 +96,12 @@ private void createInstancesTable(PipelineInstanceFilter instancesFilter) { ListSelectionModel selectionModel = instancesTable.getTable().getSelectionModel(); selectionModel.setSelectionMode(ListSelectionModel.SINGLE_SELECTION); - selectionModel.addListSelectionListener(event -> { + selectionModel.addListSelectionListener(evt -> { + if (evt.getValueIsAdjusting()) { + return; + } try { - // Ignore extra messages. - if (event.getValueIsAdjusting()) { - return; - } - - if (selectionModel.isSelectionEmpty()) { - reselectPipelineInstance(); - } else { - selectNewPipelineInstance(selectionModel.getMinSelectionIndex()); - } + selectNewPipelineInstance(selectionModel.getMinSelectionIndex()); } catch (Exception e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(parent), e); } @@ -126,34 +125,24 @@ public InstancesTableModel instancesTableModel(PipelineInstanceFilter instancesF } /** - * Reselects the correct row in the pipeline instance table after the table has been reset - * (typically by auto-refresh). + * Captures the selected row and selected pipeline instance ID when the user selects a row in + * the instances table or possibly clears the selection when the table changes. */ - private void reselectPipelineInstance() { - - // If the selected instance from before the update is still in the - // table, then select it again - int instanceModelIndex = instancesTableModel.getModelIndexOfInstance(instanceId); - - if (instanceModelIndex == -1) { - instanceId = -1; - } else { - int rowIndex = instancesTable.getTable().convertRowIndexToView(instanceModelIndex); - instancesTable.getTable().getSelectionModel().setSelectionInterval(rowIndex, rowIndex); + private void selectNewPipelineInstance(int selectedRow) { + selectedInstanceIndex = instancesTable.convertRowIndexToModel(selectedRow); + PipelineInstance pipelineInstance = selectedPipelineInstance(); + long oldInstanceId = instanceId; + instanceId = pipelineInstance != null ? pipelineInstance.getId() : -1; + if (instanceId != oldInstanceId) { + ZiggyMessenger.publish(new SelectedInstanceChangedMessage(instanceId), false); } - ZiggyMessenger.publish(new DisplayTasksForInstanceMessage(true, instanceId), false); } - /** - * Captures the selected row and selected pipeline instance ID. Executes when the user selects a - * row in the instances table. - */ - private void selectNewPipelineInstance(int selectedRow) { - selectedInstanceIndex = instancesTable.convertRowIndexToModel(selectedRow); - instanceId = instancesTableModel.getContentAtRow(selectedInstanceIndex) - .getPipelineInstance() - .getId(); - ZiggyMessenger.publish(new DisplayTasksForInstanceMessage(false, instanceId), false); + public PipelineInstance selectedPipelineInstance() { + if (selectedInstanceIndex < 0) { + return null; + } + return instancesTableModel.getContentAtRow(selectedInstanceIndex).getPipelineInstance(); } /** @@ -196,8 +185,7 @@ private JPopupMenu instancesPopupMenu() { private void displayDetails(ActionEvent evt) { try { InstanceDetailsDialog instanceDetailsDialog = new InstanceDetailsDialog( - SwingUtilities.getWindowAncestor(parent), - instancesTableModel.getContentAtRow(selectedInstanceIndex).getPipelineInstance()); + SwingUtilities.getWindowAncestor(parent), selectedPipelineInstance()); instanceDetailsDialog.setVisible(true); } catch (Throwable e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(parent), e); @@ -205,10 +193,7 @@ private void displayDetails(ActionEvent evt) { } private void displayPerformanceReport(ActionEvent evt) { - PerformanceReport report = new PerformanceReport( - instancesTableModel.getContentAtRow(selectedInstanceIndex) - .getPipelineInstance() - .getId(), + PerformanceReport report = new PerformanceReport(selectedPipelineInstance().getId(), DirectoryProperties.taskDataDir().toFile(), null); new SwingWorker() { @@ -231,9 +216,7 @@ protected void done() { private void displayAlerts(ActionEvent evt) { try { new AlertLogDialog(SwingUtilities.getWindowAncestor(parent), - instancesTableModel.getContentAtRow(selectedInstanceIndex) - .getPipelineInstance() - .getId()).setVisible(true); + selectedPipelineInstance().getId()).setVisible(true); } catch (Exception e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(parent), "Failed to display alerts", e.getMessage(), e); @@ -243,8 +226,7 @@ private void displayAlerts(ActionEvent evt) { private void displayStatistics(ActionEvent evt) { try { new InstanceStatsDialog(SwingUtilities.getWindowAncestor(parent), - instancesTableModel.getContentAtRow(selectedInstanceIndex).getPipelineInstance()) - .setVisible(true); + selectedPipelineInstance()).setVisible(true); } catch (Exception e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(parent), "Failed to retrieve performance statistics", e.getMessage(), e); @@ -254,8 +236,7 @@ private void displayStatistics(ActionEvent evt) { private void estimateCost(ActionEvent evt) { try { new InstanceCostEstimateDialog(SwingUtilities.getWindowAncestor(parent), - instancesTableModel.getContentAtRow(selectedInstanceIndex).getPipelineInstance()) - .setVisible(true); + selectedPipelineInstance()).setVisible(true); } catch (Throwable e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(parent), e); } @@ -283,15 +264,8 @@ public void loadFromDatabase() { } } - public State getStateOfInstanceWithMaxid() { - return instancesTableModel.getStateOfInstanceWithMaxid(); - } - - public PipelineInstance selectedPipelineInstance() { - if (selectedInstanceIndex < 0) { - return null; - } - return instancesTableModel.getContentAtRow(selectedInstanceIndex).getPipelineInstance(); + public State getStateOfInstanceWithMaxId() { + return instancesTableModel.getStateOfInstanceWithMaxId(); } @SuppressWarnings("serial") @@ -323,10 +297,10 @@ public InstancesTableModel(PipelineInstanceFilter filter) { @Override public void loadFromDatabase() { - new SwingWorker() { + new SwingWorker() { @Override - protected Void doInBackground() throws Exception { + protected TableUpdater doInBackground() throws Exception { // Read the pipeline instances and associated events and merge them in a // map sorted by instance ID (the key). log.trace("filter={}", filter); @@ -343,15 +317,16 @@ protected Void doInBackground() throws Exception { } // Use the sorted map to create a sorted list of container objects. + List oldInstanceEventInfoList = instanceEventInfoList; instanceEventInfoList = new ArrayList<>(instanceEventInfoById.values()); - return null; + + return new TableUpdater(oldInstanceEventInfoList, instanceEventInfoList); } @Override protected void done() { try { - get(); // check for exception - fireTableDataChanged(); + get().updateTable(InstancesTableModel.this); } catch (InterruptedException | ExecutionException e) { log.error("Can't update instances table", e); } @@ -385,7 +360,7 @@ public int getModelIndexOfInstance(long instanceId) { * * @return state of instance with max ID number */ - public State getStateOfInstanceWithMaxid() { + public State getStateOfInstanceWithMaxId() { if (instanceEventInfoList.isEmpty()) { return State.COMPLETED; } @@ -417,9 +392,9 @@ public Object getValueAt(int rowIndex, int columnIndex) { : ": " + pipelineInstance.getName()); case 2 -> ziggyEvent != null ? ziggyEvent.getEventHandlerName() : "-"; case 3 -> ziggyEvent != null ? ziggyEvent.getEventTime() - : pipelineInstance.getStartProcessingTime(); + : pipelineInstance.getCreated(); case 4 -> pipelineInstance.getState().toString(); - case 5 -> pipelineInstance.elapsedTime(); + case 5 -> pipelineInstance.getExecutionClock().toString(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); }; } @@ -451,10 +426,12 @@ private ZiggyEventOperations ziggyEventOperations() { private static class InstanceEventInfo { private PipelineInstance pipelineInstance; private ZiggyEvent ziggyEvent; + private String time; public InstanceEventInfo(PipelineInstance pipelineInstance, ZiggyEvent ziggyEvent) { this.pipelineInstance = pipelineInstance; this.ziggyEvent = ziggyEvent; + time = pipelineInstance.getExecutionClock().toString(); } public PipelineInstance getPipelineInstance() { @@ -464,5 +441,32 @@ public PipelineInstance getPipelineInstance() { public ZiggyEvent getZiggyEvent() { return ziggyEvent; } + + // When updating hashCode() and equals(), use totalHashCode() and totalEquals() with + // pipelineInstance. + @Override + public int hashCode() { + return Objects.hash(pipelineInstance.totalHashCode(), time, ziggyEvent); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + InstanceEventInfo other = (InstanceEventInfo) obj; + return pipelineInstance.totalEquals(other.pipelineInstance) + && Objects.equals(time, other.time) && Objects.equals(ziggyEvent, other.ziggyEvent); + } + + @Override + public String toString() { + return "pipelineInstance.id=" + pipelineInstance.getId() + ", time=" + time + + ", pipelineInstance.state=" + pipelineInstance.getState() + ", ziggyEvent=" + + ziggyEvent; + } } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanel.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanel.java index 0576d6d..84d63fa 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanel.java @@ -55,8 +55,7 @@ private InstancesTasksPanelAutoRefresh buildComponent() { layout.createParallelGroup().addComponent(instancesPanel).addComponent(tasksPanel)); InstancesTasksPanelAutoRefresh instancesTasksPanelAutoRefresh = new InstancesTasksPanelAutoRefresh( - instancesPanel.instancesTable(), tasksPanel.tasksTableModel(), - tasksPanel.taskStatusSummaryPanel()); + instancesPanel.instancesTable(), tasksPanel.tasksTableModel()); instancesTasksPanelAutoRefresh.start(); return instancesTasksPanelAutoRefresh; } catch (Exception e) { diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java index b3f2537..10cb094 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java @@ -3,15 +3,12 @@ import java.util.concurrent.ScheduledThreadPoolExecutor; import java.util.concurrent.TimeUnit; -import javax.swing.SwingUtilities; - import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstance.State; import gov.nasa.ziggy.services.messages.RunningPipelinesCheckRequest; -import gov.nasa.ziggy.services.messages.WorkerStatusMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.InstanceUpdateMessage; @@ -32,7 +29,6 @@ public class InstancesTasksPanelAutoRefresh implements Runnable { private ScheduledThreadPoolExecutor pool = new ScheduledThreadPoolExecutor(1); private final InstancesTable instancesTable; private final TasksTableModel tasksTableModel; - private final TaskStatusSummaryPanel taskStatusSummaryPanel; // Used to ensure that when an instance completes, a message is sent to the // worker looking for queued instances; once that message is sent, it's not sent @@ -44,17 +40,9 @@ public class InstancesTasksPanelAutoRefresh implements Runnable { private boolean priorInstancesRemaining; public InstancesTasksPanelAutoRefresh(InstancesTable instancesTable, - TasksTableModel tasksTableModel, TaskStatusSummaryPanel taskStatusSummaryPanel) { + TasksTableModel tasksTableModel) { this.instancesTable = instancesTable; this.tasksTableModel = tasksTableModel; - this.taskStatusSummaryPanel = taskStatusSummaryPanel; - - // Whenever a worker status message comes through, update the display. This will - // include both the status message that is sent on worker startup and the one sent - // on worker shutdown. - ZiggyMessenger.subscribe(WorkerStatusMessage.class, message -> { - updatePanel(); - }); } @Override @@ -64,16 +52,12 @@ public void run() { } /** - * Asks the instance and task tables to update themselves. Updates task scoreboard in the event - * dispatch thread. + * Asks the instance and task tables to update themselves. */ private void updatePanel() { instancesTable.loadFromDatabase(); tasksTableModel.loadFromDatabase(); - updateInstanceState(instancesTable.getStateOfInstanceWithMaxid()); - SwingUtilities.invokeLater(() -> { - taskStatusSummaryPanel.update(tasksTableModel); - }); + updateInstanceState(instancesTable.getStateOfInstanceWithMaxId()); } /** diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/RestartDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/RestartDialog.java index 9a54df0..d295d5b 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/RestartDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/RestartDialog.java @@ -26,7 +26,7 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; @@ -42,7 +42,7 @@ public class RestartDialog extends javax.swing.JDialog { private boolean cancelled; public RestartDialog(Window owner, - Map> supportedRunModesByPipelineTask) { + Map> supportedRunModesByPipelineTask) { super(owner, DEFAULT_MODALITY_TYPE); @@ -50,12 +50,12 @@ public RestartDialog(Window owner, setLocationRelativeTo(owner); } - private void buildComponent(Map> supportedRunModesByPipelineTask) { + private void buildComponent( + Map> supportedRunModesByPipelineTask) { setTitle("Restart failed tasks"); - getContentPane().add(createDataPanel(supportedRunModesByPipelineTask), - BorderLayout.CENTER); + getContentPane().add(createDataPanel(supportedRunModesByPipelineTask), BorderLayout.CENTER); getContentPane().add(ZiggySwingUtils.createButtonPanel(createButton(RESTART, this::restart), createButton(CANCEL, this::cancel)), BorderLayout.SOUTH); @@ -63,12 +63,12 @@ private void buildComponent(Map> supportedRunModesBy } private JScrollPane createDataPanel( - Map> supportedRunModesByPipelineTask) { + Map> supportedRunModesByPipelineTask) { return new JScrollPane(createRestartTable(supportedRunModesByPipelineTask)); } private JTable createRestartTable( - Map> supportedRunModesByPipelineTask) { + Map> supportedRunModesByPipelineTask) { restartTableModel = new RestartTableModel(supportedRunModesByPipelineTask); @@ -107,14 +107,15 @@ private void cancel(ActionEvent evt) { setVisible(false); } - // TODO Return Map and delete this comment + // TODO Return Map and delete this comment // Note that the restartAttrs variable is clobbered during the loop and the last task wins. // Given that what eventually happens is that each task gets its own restart message, and // the restart message includes the restart mode, the right thing to do is probably to - // change the logic such that a Map is returned and then passed to - // the PipelineExecutorProxy. That way we can be sure that we’re doing the right thing. + // change the logic such that a Map is returned and then + // passed to the PipelineExecutorProxy. That way we can be sure that we’re doing the right + // thing. public static RunMode restartTasks(Window owner, - Map> supportedRunModesByPipelineTask) { + Map> supportedRunModesByPipelineTask) { RestartDialog dialog = new RestartDialog(owner, supportedRunModesByPipelineTask); dialog.cancelled = false; @@ -127,14 +128,14 @@ public static RunMode restartTasks(Window owner, RestartAttributes restartAttrs = null; Map moduleMap = dialog.restartTableModel.getModuleMap(); - for (PipelineTask failedTask : supportedRunModesByPipelineTask.keySet()) { + for (PipelineTaskDisplayData failedTask : supportedRunModesByPipelineTask.keySet()) { String key = RestartAttributes.key(failedTask.getModuleName(), failedTask.getProcessingStep()); restartAttrs = moduleMap.get(key); - log.info("Set task " + failedTask.getId() + " restartMode to " - + restartAttrs.getSelectedRestartMode()); + log.info("Set task {} restartMode to {}", failedTask.getPipelineTaskId(), + restartAttrs.getSelectedRestartMode()); } return restartAttrs.getSelectedRestartMode(); } @@ -145,10 +146,11 @@ private static class RestartTableModel extends AbstractTableModel { private final List moduleList; private final Map moduleMap; - public RestartTableModel(Map> supportedRunModesByPipelineTask) { + public RestartTableModel( + Map> supportedRunModesByPipelineTask) { moduleMap = new HashMap<>(); - for (PipelineTask task : supportedRunModesByPipelineTask.keySet()) { + for (PipelineTaskDisplayData task : supportedRunModesByPipelineTask.keySet()) { String moduleName = task.getModuleName(); ProcessingStep processingStep = task.getProcessingStep(); String key = RestartAttributes.key(moduleName, processingStep); @@ -169,7 +171,7 @@ public RestartTableModel(Map> supportedRunModesByPip moduleList = new LinkedList<>(moduleMap.values()); - log.debug("moduleList.size() = " + moduleList.size()); + log.debug("moduleList.size()={}", moduleList.size()); } @Override @@ -179,7 +181,7 @@ public int getRowCount() { @Override public Object getValueAt(int rowIndex, int columnIndex) { - log.debug("getValueAt(r=" + rowIndex + ", c=" + columnIndex + ")"); + log.debug("rowIndex={}, columnIndex={}", rowIndex, columnIndex); RestartAttributes restartGroup = moduleList.get(rowIndex); @@ -218,8 +220,7 @@ public boolean isCellEditable(int rowIndex, int columnIndex) { @Override public void setValueAt(Object value, int rowIndex, int columnIndex) { - log.debug( - "setValueAt(r=" + rowIndex + ", c=" + columnIndex + ", value-=" + value + ")"); + log.debug("rowIndex={}, columnIndex={}, value={}", rowIndex, columnIndex, value); if (columnIndex != 3) { throw new IllegalArgumentException("read-only columnIndex = " + columnIndex); diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/SelectedInstanceChangedMessage.java b/src/main/java/gov/nasa/ziggy/ui/instances/SelectedInstanceChangedMessage.java new file mode 100644 index 0000000..a2329ea --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/instances/SelectedInstanceChangedMessage.java @@ -0,0 +1,25 @@ +package gov.nasa.ziggy.ui.instances; + +import gov.nasa.ziggy.services.messages.PipelineMessage; + +/** + * Message sent when selected instance changes. The currently selected instance is obtained with + * {@link #getPipelineInstanceId()}. + * + * @author PT + * @author Bill Wohler + */ +public class SelectedInstanceChangedMessage extends PipelineMessage { + + private static final long serialVersionUID = 20240913L; + + private final long pipelineInstanceId; + + public SelectedInstanceChangedMessage(long pipelineInstanceId) { + this.pipelineInstanceId = pipelineInstanceId; + } + + public long getPipelineInstanceId() { + return pipelineInstanceId; + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/SingleTaskLogDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/SingleTaskLogDialog.java index 38adc89..72ee014 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/SingleTaskLogDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/SingleTaskLogDialog.java @@ -4,6 +4,7 @@ import static gov.nasa.ziggy.ui.ZiggyGuiConstants.REFRESH; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.TO_BOTTOM; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.TO_TOP; +import static gov.nasa.ziggy.ui.util.HtmlBuilder.htmlBuilder; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; import java.awt.BorderLayout; @@ -27,7 +28,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.logging.TaskLogInformation; import gov.nasa.ziggy.services.messages.SingleTaskLogMessage; @@ -56,7 +58,8 @@ public class SingleTaskLogDialog extends javax.swing.JDialog implements Requesto private final UUID uuid = UUID.randomUUID(); - private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); public SingleTaskLogDialog(Window owner, TaskLogInformation taskLogInformation) { @@ -142,10 +145,11 @@ private void refresh() { protected String doInBackground() throws Exception { // Retrieve the task from the database - PipelineTask task = pipelineTaskOperations() - .pipelineTask(taskLogInformation.getTaskId()); - log.debug("selected task id = " + task.getId()); - taskLogLabel.setText(task.taskLabelText()); + PipelineTaskDisplayData task = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData( + pipelineTaskOperations().pipelineTask(taskLogInformation.getTaskId())); + log.debug("Selected task ID is {}", task.getPipelineTaskId()); + taskLogLabel.setText(taskLabelText(task)); // Request the log contents from the supervisor taskLogMessageCountdownLatch = new CountDownLatch(1); @@ -166,6 +170,26 @@ protected String doInBackground() throws Exception { return taskLogContents; } + /** Returns the label for a task that is used in some UI displays. */ + private String taskLabelText(PipelineTaskDisplayData pipelineTaskDisplayData) { + return htmlBuilder().appendBold("ID: ") + .append(pipelineTaskDisplayData.getPipelineInstanceId()) + .append(":") + .append(pipelineTaskDisplayData.getPipelineTaskId()) + .appendBold(" WORKER: ") + .append(pipelineTaskDisplayData.getWorkerName()) + .appendBold(" TASK: ") + .append(pipelineTaskDisplayData.getModuleName()) + .append(" [") + .append(pipelineTaskDisplayData.getBriefState()) + .append("] ") + .appendBoldColor(pipelineTaskDisplayData.getDisplayProcessingStep(), + pipelineTaskDisplayData.isError() ? "red" : "green") + .append(" ") + .appendItalic(pipelineTaskDisplayData.getExecutionClock().toString()) + .toString(); + } + @Override protected void done() { try { @@ -193,4 +217,8 @@ public UUID requestorIdentifier() { private PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TaskInfoDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/TaskInfoDialog.java index 169d257..d263f42 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TaskInfoDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TaskInfoDialog.java @@ -16,7 +16,7 @@ import javax.swing.JScrollPane; import javax.swing.LayoutStyle.ComponentPlacement; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.ui.util.HtmlBuilder; import gov.nasa.ziggy.ui.util.table.ZiggyTable; import gov.nasa.ziggy.util.dispmod.DisplayModel; @@ -28,9 +28,9 @@ */ @SuppressWarnings("serial") public class TaskInfoDialog extends javax.swing.JDialog { - private PipelineTask pipelineTask; + private PipelineTaskDisplayData pipelineTask; - public TaskInfoDialog(Window owner, PipelineTask pipelineTask) { + public TaskInfoDialog(Window owner, PipelineTaskDisplayData pipelineTask) { super(owner); this.pipelineTask = pipelineTask; @@ -54,7 +54,7 @@ private void close(ActionEvent evt) { private JPanel createDataPanel() { JLabel idLabel = boldLabel("ID:"); - JLabel idTextField = new JLabel(pipelineTask.getId() + ""); + JLabel idTextField = new JLabel(Long.toString(pipelineTask.getPipelineTaskId())); JLabel stateLabel = boldLabel("Processing step:"); JLabel stateTextField = new JLabel(HtmlBuilder.htmlBuilder() @@ -63,28 +63,26 @@ private JPanel createDataPanel() { .toString()); JLabel moduleLabel = boldLabel("Module:"); - JLabel moduleTextField = new JLabel(pipelineTask.getModuleName() + ""); + JLabel moduleTextField = new JLabel(pipelineTask.getModuleName()); JLabel uowLabel = boldLabel("Unit of work:"); - JLabel uowTextField = new JLabel(pipelineTask.uowTaskInstance().briefState()); + JLabel uowTextField = new JLabel(pipelineTask.getBriefState()); JLabel workerLabel = boldLabel("Worker:"); JLabel workerTextField = new JLabel(pipelineTask.getWorkerName()); JLabel workerHelpLabel = new JLabel("(host:thread)"); - JLabel startLabel = boldLabel("Started:"); - JLabel startTextField = new JLabel( - DisplayModel.formatDate(pipelineTask.getStartProcessingTime())); - - JLabel endLabel = boldLabel("Completed:"); - JLabel endTextField = new JLabel( - DisplayModel.formatDate(pipelineTask.getEndProcessingTime())); - JLabel createdLabel = boldLabel("Created:"); JLabel createdTextField = new JLabel(DisplayModel.formatDate(pipelineTask.getCreated())); - JLabel revisionLabel = boldLabel("Software revision:"); - JLabel revisionTextField = new JLabel(pipelineTask.getSoftwareRevision()); + JLabel durationLabel = boldLabel("Duration:"); + JLabel durationTextField = new JLabel(pipelineTask.getExecutionClock().toString()); + + JLabel ziggyRevisionLabel = boldLabel("Ziggy software revision:"); + JLabel ziggyRevisionTextField = new JLabel(pipelineTask.getZiggySoftwareRevision()); + + JLabel pipelineRevisionLabel = boldLabel("Pipeline software revision:"); + JLabel pipelineRevisionTextField = new JLabel(pipelineTask.getPipelineSoftwareRevision()); JLabel failureCountLabel = boldLabel("Failure count:"); JLabel failureCountTextField = new JLabel(Integer.toString(pipelineTask.getFailureCount())); @@ -108,10 +106,10 @@ private JPanel createDataPanel() { .addComponent(moduleLabel) .addComponent(uowLabel) .addComponent(workerLabel) - .addComponent(startLabel) - .addComponent(endLabel) + .addComponent(durationLabel) .addComponent(createdLabel) - .addComponent(revisionLabel) + .addComponent(ziggyRevisionLabel) + .addComponent(pipelineRevisionLabel) .addComponent(failureCountLabel)) .addPreferredGap(ComponentPlacement.RELATED) .addGroup(dataPanelLayout.createParallelGroup() @@ -122,10 +120,10 @@ private JPanel createDataPanel() { .addGroup(dataPanelLayout.createSequentialGroup() .addComponent(workerTextField) .addComponent(workerHelpLabel)) - .addComponent(startTextField) - .addComponent(endTextField) + .addComponent(durationTextField) .addComponent(createdTextField) - .addComponent(revisionTextField) + .addComponent(ziggyRevisionTextField) + .addComponent(pipelineRevisionTextField) .addComponent(failureCountTextField))) .addComponent(processingBreakdownTableScrollPane))); @@ -147,17 +145,17 @@ private JPanel createDataPanel() { .addComponent(workerTextField) .addComponent(workerHelpLabel)) .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(startLabel) - .addComponent(startTextField)) - .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(endLabel) - .addComponent(endTextField)) + .addComponent(durationLabel) + .addComponent(durationTextField)) .addGroup(dataPanelLayout.createParallelGroup() .addComponent(createdLabel) .addComponent(createdTextField)) .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(revisionLabel) - .addComponent(revisionTextField)) + .addComponent(ziggyRevisionLabel) + .addComponent(ziggyRevisionTextField)) + .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(pipelineRevisionLabel) + .addComponent(pipelineRevisionTextField)) .addGroup(dataPanelLayout.createParallelGroup() .addComponent(failureCountLabel) .addComponent(failureCountTextField)) @@ -173,7 +171,7 @@ private ZiggyTable createProcessingBreakdownTable() { return new ZiggyTable<>(processingBreakdownTableModel); } - public static void showTaskInfoDialog(Window owner, PipelineTask pipelineTask) { + public static void showTaskInfoDialog(Window owner, PipelineTaskDisplayData pipelineTask) { new TaskInfoDialog(owner, pipelineTask).setVisible(true); } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TaskLogInformationDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/TaskLogInformationDialog.java index efc3c24..d31c358 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TaskLogInformationDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TaskLogInformationDialog.java @@ -32,7 +32,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.logging.TaskLogInformation; import gov.nasa.ziggy.services.messages.TaskLogInformationMessage; @@ -72,11 +73,12 @@ public class TaskLogInformationDialog extends JDialog implements Requestor { private final UUID uuid = UUID.randomUUID(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); - public TaskLogInformationDialog(Window owner, PipelineTask pipelineTask) { + public TaskLogInformationDialog(Window owner, long taskId) { super(owner, DEFAULT_MODALITY_TYPE); - taskId = pipelineTask.getId(); + this.taskId = taskId; // Subscribe to TaskLogInformationMessage instances, since this panel needs to // get access to those messages. @@ -186,7 +188,7 @@ public void mouseClicked(MouseEvent evt) { int row = taskLogTable.rowAtPoint(evt.getPoint()); if (evt.getClickCount() == 2 && row != -1) { TaskLogInformation logInfo = taskLogTableModel.taskLogInformation(row); - log.info("Obtaining log file: " + logInfo.getFilename()); + log.info("Obtaining log file {}", logInfo.getFilename()); try { new SingleTaskLogDialog(TaskLogInformationDialog.this, logInfo) .setVisible(true); @@ -218,22 +220,23 @@ public void refresh() { protected Set doInBackground() throws Exception { // Get the pipeline task up-to-date information from the database. - PipelineTask task = pipelineTaskOperations().pipelineTask(taskId); - log.debug("selected task id = " + taskId); + PipelineTaskDisplayData task = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(pipelineTaskOperations().pipelineTask(taskId)); + log.debug("Selected task ID is {}", taskId); instanceText.setText(Long.toString(task.getPipelineInstanceId())); workerText.setText(task.getWorkerName()); moduleText.setText(task.getModuleName()); - uowText.setText(task.uowTaskInstance().briefState()); + uowText.setText(task.getBriefState()); processingStepText.setText(HtmlBuilder.htmlBuilder() .appendBoldColor(task.getDisplayProcessingStep(), task.isError() ? "red" : "green") .toString()); - elapsedTimeText.setText(task.elapsedTime()); + elapsedTimeText.setText(task.getExecutionClock().toString()); // Request the task log information. taskInfoRequestCountdownLatch = new CountDownLatch(1); - TaskLogInformationRequest.requestTaskLogInformation(TaskLogInformationDialog.this, - task); + ZiggyMessenger.publish(new TaskLogInformationRequest(TaskLogInformationDialog.this, + task.getPipelineTask())); // Wait for the task log to be delivered, but don't wait too long. if (!taskInfoRequestCountdownLatch.await(LOG_CONTENT_TIMEOUT_MILLIS, @@ -242,13 +245,7 @@ protected Set doInBackground() throws Exception { return null; } - // Strip the TaskLogInformation out of the message and send the message - // itself to Davy Jones' locker. - Set updatedTaskLogInformation = currentMessage - .taskLogInformation(); - currentMessage = null; - - return updatedTaskLogInformation; + return currentMessage.taskLogInformation(); } @Override @@ -275,6 +272,10 @@ private PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } + private static class TaskLogInformationTableModel extends AbstractTableModel { private static final long serialVersionUID = 20240614L; diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java b/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java index 1ac9af0..ad8949c 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java @@ -4,7 +4,7 @@ import javax.swing.table.AbstractTableModel; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.util.dispmod.ModelContentClass; import gov.nasa.ziggy.util.dispmod.TaskMetricsDisplayModel; import gov.nasa.ziggy.util.dispmod.TaskMetricsDisplayModel.ModuleTaskMetrics; @@ -19,13 +19,13 @@ public class TaskMetricsTableModel extends AbstractTableModel private TaskMetricsDisplayModel taskMetricsDisplayModel; private boolean completedTasksOnly; - public TaskMetricsTableModel(List tasks, List orderedModuleNames, - boolean completedTasksOnly) { + public TaskMetricsTableModel(List tasks, + List orderedModuleNames, boolean completedTasksOnly) { this.completedTasksOnly = completedTasksOnly; update(tasks, orderedModuleNames); } - public void update(List tasks, List orderedModuleNames) { + public void update(List tasks, List orderedModuleNames) { taskMetricsDisplayModel = new TaskMetricsDisplayModel(tasks, orderedModuleNames, completedTasksOnly); fireTableDataChanged(); diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TaskStatusSummaryPanel.java b/src/main/java/gov/nasa/ziggy/ui/instances/TaskStatusSummaryPanel.java index d941dd9..c115df4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TaskStatusSummaryPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TaskStatusSummaryPanel.java @@ -1,11 +1,21 @@ package gov.nasa.ziggy.ui.instances; import java.awt.BorderLayout; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutionException; import javax.swing.JScrollPane; +import javax.swing.SwingWorker; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.TaskCounts.Counts; +import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; +import gov.nasa.ziggy.ui.util.table.TableUpdater; import gov.nasa.ziggy.ui.util.table.ZiggyTable; import gov.nasa.ziggy.util.dispmod.TaskSummaryDisplayModel; @@ -14,6 +24,8 @@ */ @SuppressWarnings("serial") public class TaskStatusSummaryPanel extends javax.swing.JPanel { + private static final Logger log = LoggerFactory.getLogger(TaskStatusSummaryPanel.class); + private TaskSummaryTableModel taskSummaryTableModel; public TaskStatusSummaryPanel() { @@ -23,11 +35,13 @@ public TaskStatusSummaryPanel() { setLayout(new BorderLayout()); setPreferredSize(new java.awt.Dimension(400, 112)); add(new JScrollPane(ziggyTable.getTable()), BorderLayout.CENTER); + + ZiggyMessenger.subscribe(TasksUpdatedMessage.class, + message -> taskSummaryTableModel.update(message.getTasksTableModel())); } /** - * Updates the task status summary ("scoreboard") panel. This method must be called from the - * event dispatch thread only. + * Updates the task status summary ("scoreboard") panel. */ public void update(TasksTableModel tasksTableModel) { taskSummaryTableModel.update(tasksTableModel); @@ -35,14 +49,38 @@ public void update(TasksTableModel tasksTableModel) { private static class TaskSummaryTableModel extends AbstractZiggyTableModel { private final TaskSummaryDisplayModel taskSummaryDisplayModel = new TaskSummaryDisplayModel(); - - public TaskSummaryTableModel() { - } + private List counts; public void update(TasksTableModel tasksTableModel) { - taskSummaryDisplayModel.update(tasksTableModel.getTaskStates()); + new SwingWorker() { + + @Override + protected TableUpdater doInBackground() { + List oldCounts = counts; + TaskCounts taskCounts = tasksTableModel.getTaskStates(); + taskSummaryDisplayModel.update(taskCounts); + counts = taskCountsToCountList(taskCounts); + return new TableUpdater(oldCounts, counts); + } + + @Override + protected void done() { + try { + get().updateTable(TaskSummaryTableModel.this); + } catch (InterruptedException | ExecutionException e) { + log.error("Can't update task summary table", e); + } + } + }.execute(); + } - fireTableDataChanged(); + private List taskCountsToCountList(TaskCounts taskCounts) { + List counts = new ArrayList<>(); + for (String moduleName : taskCounts.getModuleNames()) { + counts.add(taskCounts.getModuleCounts().get(moduleName)); + } + counts.add(taskCounts.getTotalCounts()); + return counts; } @Override diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TasksPanel.java b/src/main/java/gov/nasa/ziggy/ui/instances/TasksPanel.java index 88474b2..cb24031 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TasksPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TasksPanel.java @@ -7,9 +7,8 @@ import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createPopupMenu; import java.awt.event.ActionEvent; -import java.util.ArrayList; +import java.awt.event.MouseListener; import java.util.HashMap; -import java.util.List; import java.util.Map; import javax.swing.GroupLayout; @@ -21,15 +20,14 @@ import javax.swing.LayoutStyle.ComponentPlacement; import javax.swing.ListSelectionModel; import javax.swing.SwingUtilities; -import javax.swing.SwingWorker; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.MessageUtils; @@ -53,74 +51,34 @@ public class TasksPanel extends JPanel { @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(TasksPanel.class); - private TaskStatusSummaryPanel taskStatusSummaryPanel; private ZiggyTable tasksTable; private TasksTableModel tasksTableModel; private long currentInstanceId = -1; - private Map selectedTaskIdByRow = new HashMap<>(); + private Map selectedTasksByRow = new HashMap<>(); - private final PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); + private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); public TasksPanel() { buildComponent(); // Subscribe to the messages that indicate that something has potentially changed // with the instance panel's instance selection. - ZiggyMessenger.subscribe(DisplayTasksForInstanceMessage.class, + ZiggyMessenger.subscribe(SelectedInstanceChangedMessage.class, this::displayTasksForInstance); ZiggyMessenger.subscribe(InvalidateConsoleModelsMessage.class, this::invalidateModel); } - private void displayTasksForInstance(DisplayTasksForInstanceMessage message) { - new SwingWorker() { - - @Override - protected Void doInBackground() { - long instanceId = message.getPipelineInstanceId(); - - if (message.isReselect() && instanceId == -1) { - currentInstanceId = instanceId; - - // If the message comes from a reselect action, we only need to do anything - // if the reselect resulted in no selected instance. - tasksTableModel.setPipelineInstance(null); - tasksTableModel.loadFromDatabase(false); - clearSelectedTasks(); - } else if (!message.isReselect()) { - - // If the message comes from a user clicking on a row in the table, we - // need to see whether this is a genuinely new instance, or if the user - // just clicked on the instance that was already selected. - PipelineInstance selectedInstance = null; - if (instanceId != currentInstanceId) { - selectedInstance = pipelineInstanceOperations() - .pipelineInstance(instanceId); - currentInstanceId = instanceId; - } - boolean genuinelyNewInstance = selectedInstance != null - ? tasksTableModel.updatePipelineInstance(selectedInstance) - : false; - tasksTableModel.loadFromDatabase(false); - - // If the change is that a genuinely new instance was selected, clear the tasks - // table selections as they are no longer valid. - if (genuinelyNewInstance) { - SwingUtilities.invokeLater( - () -> tasksTable.getTable().getSelectionModel().clearSelection()); - clearSelectedTasks(); - } - } - return null; - } - - @Override - protected void done() { - // No matter what happened, update the summary display. - updateSummaryPanel(); - } - }.execute(); + private void displayTasksForInstance(SelectedInstanceChangedMessage message) { + if (message.getPipelineInstanceId() == currentInstanceId) { + return; + } + tasksTable.getTable().getSelectionModel().clearSelection(); + ZiggySwingUtils.flushEventDispatchThread(); + selectedTasksByRow.clear(); + currentInstanceId = message.getPipelineInstanceId(); + tasksTableModel.updatePipelineInstanceId(currentInstanceId); } private void invalidateModel(InvalidateConsoleModelsMessage message) { @@ -128,9 +86,9 @@ private void invalidateModel(InvalidateConsoleModelsMessage message) { } private void buildComponent() { - JLabel instance = boldLabel("Pipeline tasks", LabelType.HEADING1); + JLabel tasks = boldLabel("Pipeline tasks", LabelType.HEADING1); - taskStatusSummaryPanel = new TaskStatusSummaryPanel(); + TaskStatusSummaryPanel taskStatusSummaryPanel = new TaskStatusSummaryPanel(); tasksTableModel = new TasksTableModel(); tasksTable = createTasksTable(tasksTableModel); @@ -141,12 +99,12 @@ private void buildComponent() { setLayout(layout); layout.setHorizontalGroup(layout.createParallelGroup() - .addComponent(instance) + .addComponent(tasks) .addComponent(taskStatusSummaryPanel) .addComponent(tasksTableScrollPane)); layout.setVerticalGroup(layout.createSequentialGroup() - .addComponent(instance) + .addComponent(tasks) .addPreferredGap(ComponentPlacement.UNRELATED) .addComponent(taskStatusSummaryPanel, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) @@ -162,68 +120,30 @@ private ZiggyTable createTasksTable(TasksTableModel tasksTableMode ListSelectionModel selectionModel = tasksTable.getTable().getSelectionModel(); selectionModel.setSelectionMode(ListSelectionModel.MULTIPLE_INTERVAL_SELECTION); - selectionModel.addListSelectionListener(e -> { - ListSelectionModel lsm = (ListSelectionModel) e.getSource(); - - if (!lsm.getValueIsAdjusting()) { - - if (lsm.isSelectionEmpty()) { - reselectTaskRows(lsm); - } else { - captureSelectedTaskRows(lsm); - } + selectionModel.addListSelectionListener(evt -> { + if (evt.getValueIsAdjusting()) { + return; } + captureSelectedTaskRows(selectionModel); }); return tasksTable; } /** - * Repopulate the selected task rows after the table has been reset. If the table has been reset - * because the user selected a different pipeline instance, then the tasks that were selected - * are no longer present and the cache of selected rows / tasks needs to be cleared. + * Capture the selected task table rows and the corresponding tasks. */ - private void reselectTaskRows(ListSelectionModel lsm) { - - // If there were no selected tasks in the first place, return. - if (selectedTaskIdByRow.isEmpty()) { - return; - } - - // Find the model indices of the selected task IDs. Here we use a List of - // task IDs so that the order of the task IDs and the model IDs match one another. - List selectedTaskIds = new ArrayList<>(selectedTaskIdByRow.values()); - List taskModelIds = tasksTableModel.getModelIndicesOfTasks(selectedTaskIds); - selectedTaskIdByRow.clear(); - - // If all the tasks that had been selected before are now gone from the table, return. - if (taskModelIds.isEmpty()) { - return; - } - - // Convert the model indices to view indices and repopulate the Map. - for (int listIndex = 0; listIndex < taskModelIds.size(); listIndex++) { - int taskModelId = taskModelIds.get(listIndex); - long taskId = selectedTaskIds.get(listIndex); - int rowId = tasksTable.getTable().convertRowIndexToView(taskModelId); - lsm.addSelectionInterval(rowId, rowId); - selectedTaskIdByRow.put(rowId, taskId); - } - } - - /** - * Capture the selected task table rows and the corresponding task IDs. - */ - private void captureSelectedTaskRows(ListSelectionModel lsm) { + private void captureSelectedTaskRows(ListSelectionModel selectionModel) { // Clear the cache of selected rows / tasks. - selectedTaskIdByRow.clear(); + selectedTasksByRow.clear(); - // Capture the selected row index and task ID in the cache. - for (int index : lsm.getSelectedIndices()) { + // Capture the selected row index and task in the cache. + for (int index : selectionModel.getSelectedIndices()) { int rowModel = tasksTable.convertRowIndexToModel(index); - long taskId = tasksTableModel.getContentAtRow(rowModel).getId(); - selectedTaskIdByRow.put(index, taskId); + PipelineTask pipelineTask = pipelineTaskOperations() + .pipelineTask(tasksTableModel.getContentAtRow(rowModel).getPipelineTaskId()); + selectedTasksByRow.put(index, pipelineTask); } } @@ -241,13 +161,21 @@ private void createTasksTablePopupMenu() { JPopupMenu tasksPopupMenu = createPopupMenu(detailsMenuItem, retrieveLogInfoMenuItem, MENU_SEPARATOR, restartFailedTasksMenuItem, MENU_SEPARATOR, haltTasksMenuItem); + // Remove the existing table mouse listener as it clears the selection with Control-Click on + // the Mac rather than preserving the selection before it displays a menu. Perform selection + // ourselves using ZiggySwingUtils.adjustSelection(). See below. + for (MouseListener l : tasksTable.getTable().getMouseListeners()) { + if (l.getClass().getName().equals("javax.swing.plaf.basic.BasicTableUI$Handler")) { + tasksTable.getTable().removeMouseListener(l); + } + } tasksTable.getTable().addMouseListener(new java.awt.event.MouseAdapter() { @Override public void mousePressed(java.awt.event.MouseEvent e) { + ZiggySwingUtils.adjustSelection(tasksTable.getTable(), e); if (e.isPopupTrigger()) { - ZiggySwingUtils.adjustSelection(tasksTable.getTable(), e); - detailsMenuItem.setEnabled(selectedTaskIdByRow.size() == 1); - retrieveLogInfoMenuItem.setEnabled(selectedTaskIdByRow.size() == 1); + detailsMenuItem.setEnabled(selectedTasksByRow.size() == 1); + retrieveLogInfoMenuItem.setEnabled(selectedTasksByRow.size() == 1); tasksPopupMenu.show(tasksTable.getTable(), e.getX(), e.getY()); } } @@ -255,7 +183,7 @@ public void mousePressed(java.awt.event.MouseEvent e) { } private void showDetails(ActionEvent evt) { - PipelineTask selectedTask = selectedTask(); + PipelineTaskDisplayData selectedTask = selectedTask(); if (selectedTask != null) { try { TaskInfoDialog.showTaskInfoDialog(SwingUtilities.getWindowAncestor(this), @@ -267,12 +195,12 @@ private void showDetails(ActionEvent evt) { } private void retrieveLogInfo(ActionEvent evt) { - PipelineTask selectedTask = selectedTask(); + PipelineTaskDisplayData selectedTask = selectedTask(); if (selectedTask != null) { try { - new TaskLogInformationDialog(SwingUtilities.getWindowAncestor(this), selectedTask) - .setVisible(true); + new TaskLogInformationDialog(SwingUtilities.getWindowAncestor(this), + selectedTask.getPipelineTaskId()).setVisible(true); } catch (PipelineException e) { MessageUtils.showError(SwingUtilities.getWindowAncestor(this), e); } @@ -280,53 +208,38 @@ private void retrieveLogInfo(ActionEvent evt) { } private void restartTasks(ActionEvent evt) { - if (selectedTaskIdByRow.isEmpty()) { + if (selectedTasksByRow.isEmpty()) { return; } new TaskRestarter().restartTasks(SwingUtilities.getWindowAncestor(this), - tasksTableModel.getPipelineInstance(), selectedTaskIdByRow.values()); + tasksTableModel.getPipelineInstance(), selectedTasksByRow.values()); } private void haltTasks(ActionEvent evt) { - if (selectedTaskIdByRow.isEmpty()) { + if (selectedTasksByRow.isEmpty()) { return; } new TaskHalter().haltTasks(SwingUtilities.getWindowAncestor(this), - tasksTableModel.getPipelineInstance(), selectedTaskIdByRow.values()); + tasksTableModel.getPipelineInstance(), selectedTasksByRow.values()); } - private PipelineTask selectedTask() { - if (selectedTaskIdByRow.size() != 1) { + private PipelineTaskDisplayData selectedTask() { + if (selectedTasksByRow.size() != 1) { return null; } - int selectedIndex = selectedTaskIdByRow.keySet().iterator().next(); + int selectedIndex = selectedTasksByRow.keySet().iterator().next(); int selectedModelRow = tasksTable.convertRowIndexToModel(selectedIndex); return tasksTableModel.getContentAtRow(selectedModelRow); } - /** - * Allows the instances panel to clear the selected tasks when necessary. - */ - void clearSelectedTasks() { - selectedTaskIdByRow.clear(); - } - public TasksTableModel tasksTableModel() { return tasksTableModel; } - public TaskStatusSummaryPanel taskStatusSummaryPanel() { - return taskStatusSummaryPanel; - } - - private void updateSummaryPanel() { - taskStatusSummaryPanel.update(tasksTableModel); - } - - private PipelineInstanceOperations pipelineInstanceOperations() { - return pipelineInstanceOperations; + private PipelineTaskOperations pipelineTaskOperations() { + return pipelineTaskOperations; } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TasksTableModel.java b/src/main/java/gov/nasa/ziggy/ui/instances/TasksTableModel.java index 5b60d2e..fc3024a 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TasksTableModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TasksTableModel.java @@ -1,9 +1,10 @@ package gov.nasa.ziggy.ui.instances; -import java.util.ArrayList; import java.util.LinkedList; import java.util.List; +import java.util.Objects; import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; import javax.swing.SwingWorker; @@ -11,15 +12,19 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.models.DatabaseModel; +import gov.nasa.ziggy.ui.util.table.TableUpdater; import gov.nasa.ziggy.util.dispmod.TasksDisplayModel; @SuppressWarnings("serial") -public class TasksTableModel extends AbstractZiggyTableModel +public class TasksTableModel extends AbstractZiggyTableModel implements DatabaseModel { private static final Logger log = LoggerFactory.getLogger(TasksTableModel.class); @@ -27,61 +32,46 @@ public class TasksTableModel extends AbstractZiggyTableModel /** Preferred column widths. */ public static final int[] COLUMN_WIDTHS = TasksDisplayModel.COLUMN_WIDTHS; + private long pipelineInstanceId; private PipelineInstance pipelineInstance; - private List tasks = new LinkedList<>(); - - // Used to store member values when the instance is TEMPORARILY set to null - private List stashedTasks; - long instanceIdForStashedTaskInfo = -1L; - + private List tasks = new LinkedList<>(); private TasksDisplayModel tasksDisplayModel = new TasksDisplayModel(); private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); - - private enum UpdateMode { - SAME_INSTANCE, REPLACE_TEMP_NULL, FULL_UPDATE - } - - public TasksTableModel() { - stashedTasks = tasks; - } + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + private final PipelineInstanceOperations pipelineInstanceOperations = new PipelineInstanceOperations(); @Override public void loadFromDatabase() { - loadFromDatabase(true); - } - - public void loadFromDatabase(boolean forceUpdate) { - new SwingWorker() { + new SwingWorker() { @Override - protected Void doInBackground() throws Exception { - if (pipelineInstance == null) { - tasks = new LinkedList<>(); - } else { - UpdateMode updateMode = getUpdateMode(forceUpdate); - switch (updateMode) { - case SAME_INSTANCE: - break; - case REPLACE_TEMP_NULL: - tasks = stashedTasks; - break; - case FULL_UPDATE: - tasks = pipelineTaskOperations().pipelineTasks(pipelineInstance, true); - break; - default: - throw new IllegalStateException( - "Unsupported update mode " + updateMode.toString()); + protected TableUpdater doInBackground() throws Exception { + List pipelineTaskDisplayData; + if (pipelineInstanceId > 0) { + if (pipelineInstance == null + || pipelineInstance.getId() != pipelineInstanceId) { + pipelineInstance = pipelineInstanceOperations() + .pipelineInstance(pipelineInstanceId); } + pipelineTaskDisplayData = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(pipelineInstance); + } else { + pipelineInstance = null; + pipelineTaskDisplayData = new LinkedList<>(); } - return null; + tasksDisplayModel.update(pipelineTaskDisplayData); + List oldTasks = tasks; + tasks = pipelineTaskDisplayData.stream() + .map(TaskTimeInfo::new) + .collect(Collectors.toList()); + return new TableUpdater(oldTasks, tasks); } @Override protected void done() { try { - get(); // check for exception - tasksDisplayModel = new TasksDisplayModel(tasks); - fireTableDataChanged(); + get().updateTable(TasksTableModel.this); + ZiggyMessenger.publish(new TasksUpdatedMessage(TasksTableModel.this), false); } catch (InterruptedException | ExecutionException e) { log.error("Could not load pipeline tasks or attributes", e); } @@ -89,38 +79,6 @@ protected void done() { }.execute(); } - /** - * Determines the update mode to use when refreshing task and task attribute information. This - * depends on the values of the current instance, the last instance, the stashed instance, and - * the forceUpdate flag: - *

    - *
  1. If the current instance is the same as the last instance, no update is needed. - *
  2. If the last instance was null, but the current instance matches the stashed instance, we - * can refresh the task information from the stashed information. - *
  3. If the current instance is different from the last non-null instance (which is either the - * last instance or the stashed instance), a full update is needed. - *
  4. If the fullUpdate flag is true, then a full update is needed regardless of these other - * considerations. - *
- */ - private UpdateMode getUpdateMode(boolean forceUpdate) { - - UpdateMode updateMode = UpdateMode.FULL_UPDATE; - - if (tasks.isEmpty() && instanceIdForStashedTaskInfo == pipelineInstance.getId()) { - updateMode = UpdateMode.REPLACE_TEMP_NULL; - } else if (!tasks.isEmpty() && instanceIdForStashedTaskInfo == pipelineInstance.getId()) { - updateMode = UpdateMode.SAME_INSTANCE; - } else if (instanceIdForStashedTaskInfo != pipelineInstance.getId()) { - updateMode = UpdateMode.FULL_UPDATE; - } - - if (forceUpdate) { - updateMode = UpdateMode.FULL_UPDATE; - } - return updateMode; - } - @Override public int getRowCount() { return tasksDisplayModel.getRowCount(); @@ -141,76 +99,79 @@ public String getColumnName(int column) { return tasksDisplayModel.getColumnName(column); } - public PipelineInstance getPipelineInstance() { - return pipelineInstance; + @Override + public PipelineTaskDisplayData getContentAtRow(int row) { + return tasks.get(row).getPipelineTaskDisplayData(); } - /** - * Sets the pipeline instance and determines whether it is a genuinely new instance. A genuinely - * new instance is one that is different from the current instance (if that instance is - * non-null), or different from the stashed instance (if the current instance is null). This - * information is used to decide what to do about the selected row in the tasks table. - */ - public boolean updatePipelineInstance(PipelineInstance pipelineInstance) { - boolean genuinelyNewInstance = true; - if (this.pipelineInstance == null - && pipelineInstance.getId() == instanceIdForStashedTaskInfo) { - genuinelyNewInstance = false; - } - if (this.pipelineInstance != null && this.pipelineInstance.equals(pipelineInstance)) { - genuinelyNewInstance = false; - } - setPipelineInstance(pipelineInstance); - return genuinelyNewInstance; + @Override + public Class tableModelContentClass() { + return PipelineTaskDisplayData.class; } - public void setPipelineInstance(PipelineInstance pipelineInstance) { - - // Stash current pipeline instance information. Don't bother to do this if the - // current pipeline instance is null, as we want to stash the most recent non-null - // state (i.e., if we get 10 updates to null in a row, we still want the stashed - // information to be from the non-null update that came before those 10 null updates). - if (pipelineInstance != null) { - stashTaskInfo(); - } - this.pipelineInstance = pipelineInstance; + public PipelineInstance getPipelineInstance() { + return pipelineInstance; } - public List getModelIndicesOfTasks(List taskIds) { - List modelIndices = new ArrayList<>(); - for (int i = 0; i < tasks.size(); i++) { - if (taskIds.contains(tasks.get(i).getId())) { - modelIndices.add(i); - } + public void updatePipelineInstanceId(long instanceId) { + if (instanceId == pipelineInstanceId) { + return; } - return modelIndices; - } - - /** - * Stores the current task information state and the current instance ID. This makes it possible - * to use stashed information to update the object when a new pipeline instance is supplied but - * the new instance ID is the same as the last non-null instance ID. - */ - private void stashTaskInfo() { - stashedTasks = tasks; - instanceIdForStashedTaskInfo = pipelineInstance != null ? pipelineInstance.getId() : -1L; + pipelineInstanceId = instanceId; + loadFromDatabase(); } public TaskCounts getTaskStates() { return tasksDisplayModel.getTaskCounts(); } - @Override - public PipelineTask getContentAtRow(int row) { - return tasks.get(row); + PipelineTaskOperations pipelineTaskOperations() { + return pipelineTaskOperations; } - @Override - public Class tableModelContentClass() { - return PipelineTask.class; + PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; } - PipelineTaskOperations pipelineTaskOperations() { - return pipelineTaskOperations; + PipelineInstanceOperations pipelineInstanceOperations() { + return pipelineInstanceOperations; + } + + private static class TaskTimeInfo { + private PipelineTaskDisplayData pipelineTaskDisplayData; + private String time; + + public TaskTimeInfo(PipelineTaskDisplayData pipelineTaskDisplayData) { + this.pipelineTaskDisplayData = pipelineTaskDisplayData; + time = getPipelineTaskDisplayData().getExecutionClock().toString(); + } + + public PipelineTaskDisplayData getPipelineTaskDisplayData() { + return pipelineTaskDisplayData; + } + + @Override + public int hashCode() { + return Objects.hash(pipelineTaskDisplayData, time); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + TaskTimeInfo other = (TaskTimeInfo) obj; + return Objects.equals(pipelineTaskDisplayData, other.pipelineTaskDisplayData) + && Objects.equals(time, other.time); + } + + @Override + public String toString() { + return "pipelineTask=" + getPipelineTaskDisplayData().toString() + "(" + + getPipelineTaskDisplayData().getProcessingStep() + "), time=" + time; + } } } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TasksUpdatedMessage.java b/src/main/java/gov/nasa/ziggy/ui/instances/TasksUpdatedMessage.java new file mode 100644 index 0000000..4b1669c --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TasksUpdatedMessage.java @@ -0,0 +1,25 @@ +package gov.nasa.ziggy.ui.instances; + +import gov.nasa.ziggy.services.messages.PipelineMessage; + +/** + * Message sent to panels that depend on the task panel's data to indicate that the data in the task + * panel has been updated. + * + * @author Bill Wohler + */ + +public class TasksUpdatedMessage extends PipelineMessage { + + private static final long serialVersionUID = 20240917L; + + private final TasksTableModel tasksTableModel; + + public TasksUpdatedMessage(TasksTableModel tasksTableModel) { + this.tasksTableModel = tasksTableModel; + } + + public TasksTableModel getTasksTableModel() { + return tasksTableModel; + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/DatabaseMetricsValueSource.java b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/DatabaseMetricsValueSource.java index b27e4e9..e04cec5 100644 --- a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/DatabaseMetricsValueSource.java +++ b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/DatabaseMetricsValueSource.java @@ -26,7 +26,8 @@ public Map> metricValues( Map> rv = Maps .newHashMapWithExpectedSize(selectedMetricTypes.size()); for (MetricType type : selectedMetricTypes) { - List values = metricsOperations().metricValues(type, windowStart, windowEnd); + List values = metricsOperations().metricValues(type, windowStart, + windowEnd); rv.put(type, values); } return rv; diff --git a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/MetricsChartPanel.java b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/MetricsChartPanel.java index b2232f9..860c148 100644 --- a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/MetricsChartPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/MetricsChartPanel.java @@ -71,12 +71,12 @@ public void clearChart() { public void addMetric(String name, Collection metricList, int binSizeMillis) { if (metricList == null) { - log.error("sampleList is null"); + log.error("Parameter metricList is null"); return; } if (metricList.size() == 0) { - log.error("sampleList is empty"); + log.error("Parameter metricList is empty"); return; } @@ -101,7 +101,7 @@ public void addMetric(String name, Collection metricList, int binSi samples.bin(binSizeMillis); } - log.debug("adding series[" + name + "] to dataset, #samples = " + samples.size()); + log.debug("Adding series[{}] to dataset, {} samples", name, samples.size()); dataset.addSeries(samples.asTimeSeries(name)); } } diff --git a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/SampleList.java b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/SampleList.java index 29309cb..c458c46 100644 --- a/src/main/java/gov/nasa/ziggy/ui/metrilyzer/SampleList.java +++ b/src/main/java/gov/nasa/ziggy/ui/metrilyzer/SampleList.java @@ -96,12 +96,9 @@ public void addSample(long timestamp, double value) { samples.add(newSample); } - /** - * @param binSizeMillis - */ public void bin(long binSizeMillis) { if (samples.size() == 0) { - log.warn("sample list empty! No bin for you!"); + log.warn("Not binning an empty sample list"); return; } @@ -115,21 +112,21 @@ public void bin(long binSizeMillis) { double sum = 0.0; double count = 0.0; - log.info("START binning, input set size=" + samples.size()); + log.info("Binning {} samples", samples.size()); while (thisBinStart < lastSampleTime) { if (currentSample.time >= nextBinStart) { - // end of bin - log.debug("end of bin, count=" + count); + // End of bin. + log.debug("End of bin, count={}", count); if (count > 0.0) { - // at least one sample in this bin, store the average - log.debug("adding sample @" + new Date(currentBinMid) + " = " + sum / count); + // At least one sample in this bin, store the average. + log.debug("Adding sample @{}={}", new Date(currentBinMid), sum / count); newList.add(new Sample(currentBinMid, sum / count)); } thisBinStart = nextBinStart; nextBinStart = thisBinStart + binSizeMillis; currentBinMid = (thisBinStart + nextBinStart) / 2; - log.debug("new bin start = " + new Date(thisBinStart)); + log.debug("New bin start={}", new Date(thisBinStart)); sum = 0.0; count = 0.0; } else { @@ -143,19 +140,19 @@ public void bin(long binSizeMillis) { } // for (Sample sample : samples) { -// log.debug("sample.time = " + new Date(sample.time)); +// log.debug("Sample time is {}", new Date(sample.time)); // if (sample.time >= nextBinStart) { -// // end of bin -// log.debug("end of bin, count=" + count); +// // End of bin. +// log.debug("End of bin, count={}", count); // if (count > 0.0) { -// // at least one sample in this bin, store the average -// log.debug("adding sample @" + new Date(currentBinMid) + " = " + sum / count); +// // At least one sample in this bin, store the average. +// log.debug("Adding sample @{}={}",sum / count, new Date(currentBinMid)); // newList.add(new Sample(currentBinMid, sum / count)); // } // thisBinStart = nextBinStart; // nextBinStart = thisBinStart + binSizeMillis; // currentBinMid = (thisBinStart + nextBinStart) / 2; -// log.debug("new bin start = " + new Date(thisBinStart)); +// log.debug("New bin start={}", new Date(thisBinStart)); // sum = 0.0; // count = 0.0; // } @@ -164,7 +161,8 @@ public void bin(long binSizeMillis) { // count += 1.0; // } - log.info("END binning, output set size=" + newList.size()); + log.info("Binning {} samples...done, output set size is {}", samples.size(), + newList.size()); samples = newList; } diff --git a/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java b/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java index a4d852e..02efec4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java @@ -9,15 +9,10 @@ import java.awt.BorderLayout; import java.awt.Window; import java.awt.event.ActionEvent; -import java.util.Collections; -import java.util.LinkedList; -import java.util.List; -import java.util.Set; import java.util.concurrent.ExecutionException; -import javax.swing.DefaultComboBoxModel; +import javax.swing.BorderFactory; import javax.swing.GroupLayout; -import javax.swing.JComboBox; import javax.swing.JLabel; import javax.swing.JPanel; import javax.swing.JScrollPane; @@ -29,13 +24,10 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineModuleExecutionResources; import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; import gov.nasa.ziggy.ui.ZiggyGuiConstants; -import gov.nasa.ziggy.ui.util.ClasspathUtils; import gov.nasa.ziggy.ui.util.MessageUtils; /** @@ -43,14 +35,12 @@ * @author Bill Wohler */ public class EditModuleDialog extends javax.swing.JDialog { - private static final long serialVersionUID = 20240614L; + private static final long serialVersionUID = 20241002L; private static final Logger log = LoggerFactory.getLogger(EditModuleDialog.class); private PipelineModuleDefinition module; private PipelineModuleExecutionResources executionResources; - private JTextArea descText; - private JComboBox> implementingClassComboBox; private JTextField exeTimeoutText; private JTextField minMemoryText; @@ -83,15 +73,15 @@ private JPanel createDataPanel() { JLabel nameText = new JLabel(module.getName()); JLabel desc = boldLabel("Description"); - descText = new JTextArea(); - descText.setRows(4); + JTextArea descText = new JTextArea(module.getDescription()); descText.setLineWrap(true); descText.setWrapStyleWord(true); - descText.setText(module.getDescription()); + descText.setEditable(false); JScrollPane descScrollPane = new JScrollPane(descText); + descScrollPane.setBorder(BorderFactory.createEmptyBorder()); JLabel implementingClass = boldLabel("Implementing class"); - implementingClassComboBox = createImplementingClassComboBox(); + JLabel implementingClassText = new JLabel(module.getPipelineModuleClass().toString()); JLabel exeTimeout = boldLabel("Executable timeout (seconds)"); exeTimeoutText = new JTextField(ZiggyGuiConstants.LOADING); @@ -112,8 +102,7 @@ private JPanel createDataPanel() { .addComponent(desc) .addComponent(descScrollPane) .addComponent(implementingClass) - .addComponent(implementingClassComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addComponent(implementingClassText) .addComponent(exeTimeout) .addComponent(exeTimeoutText, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) @@ -129,8 +118,7 @@ private JPanel createDataPanel() { .addComponent(descScrollPane) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(implementingClass) - .addComponent(implementingClassComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addComponent(implementingClassText) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(exeTimeout) .addComponent(exeTimeoutText, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, @@ -142,63 +130,12 @@ private JPanel createDataPanel() { return dataPanel; } - private JComboBox> createImplementingClassComboBox() { - - DefaultComboBoxModel> implementingClassComboBoxModel = new DefaultComboBoxModel<>(); - JComboBox> implementingClassComboBox = new JComboBox<>( - implementingClassComboBoxModel); - - try { - Set> detectedClasses = ClasspathUtils - .scanFully(PipelineModule.class); - List> wrapperList = new LinkedList<>(); - - for (Class clazz : detectedClasses) { - try { - wrapperList.add(new ClassWrapper<>(clazz)); - } catch (Exception ignore) { - } - } - - Collections.sort(wrapperList); - - int selectedIndex = -1; - int index = 0; - - for (ClassWrapper classWrapper : wrapperList) { - implementingClassComboBoxModel.addElement(classWrapper); - - ClassWrapper implementingClass = module.getPipelineModuleClass(); - if (implementingClass != null - && classWrapper.getClazz().equals(implementingClass.getClazz())) { - selectedIndex = index; - } - index++; - } - - if (selectedIndex != -1) { - implementingClassComboBox.setSelectedIndex(selectedIndex); - } - } catch (Exception e) { - MessageUtils.showError(this, e); - } - return implementingClassComboBox; - } - private void save(ActionEvent evt) { new SwingWorker() { @Override protected Void doInBackground() throws Exception { - @SuppressWarnings("unchecked") - ClassWrapper selectedImplementingClass = (ClassWrapper) implementingClassComboBox - .getSelectedItem(); - - module.setDescription(descText.getText()); - module.setPipelineModuleClass(selectedImplementingClass); - module = pipelineModuleDefinitionOperations().merge(module); - executionResources.setExeTimeoutSeconds(toInt(exeTimeoutText.getText(), 0)); executionResources.setMinMemoryMegabytes(toInt(minMemoryText.getText(), 0)); executionResources = pipelineModuleDefinitionOperations().merge(executionResources); diff --git a/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java b/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java index a29134a..661dcac 100644 --- a/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java @@ -3,10 +3,8 @@ import java.util.Date; import java.util.LinkedList; import java.util.List; -import java.util.Set; import java.util.concurrent.ExecutionException; -import javax.swing.JOptionPane; import javax.swing.SwingUtilities; import javax.swing.SwingWorker; @@ -34,8 +32,6 @@ public class ViewEditModuleLibraryPanel extends AbstractViewEditPanel optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.NEW, - OptionalViewEditFunction.RENAME); - } - @Override protected void refresh() { try { @@ -63,104 +53,11 @@ protected void refresh() { } } - @Override - protected void create() { - - String newModuleName = readModuleName("Enter the name for the new module definition", - "New pipeline module definition"); - if (newModuleName == null) { - return; - } - - showEditDialog(new PipelineModuleDefinition(newModuleName)); - } - @Override protected void edit(int row) { showEditDialog(ziggyTable.getContentAtViewRow(row)); } - @Override - protected void rename(int row) { - - String newModuleName = readModuleName("Enter the new name for this module definition", - "Rename module definition"); - if (newModuleName == null) { - return; - } - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - pipelineModuleDefinitionOperations().rename(ziggyTable.getContentAtViewRow(row), - newModuleName); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (InterruptedException | ExecutionException e) { - MessageUtils.showError(ViewEditModuleLibraryPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void delete(int row) { - - PipelineModuleDefinition module = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete module '" + module.getName() + "'?"); - if (choice != JOptionPane.YES_OPTION) { - return; - } - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - pipelineModuleDefinitionOperations().delete(module); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (InterruptedException | ExecutionException e) { - MessageUtils.showError(ViewEditModuleLibraryPanel.this, e); - } - - super.done(); - } - }.execute(); - } - - private String readModuleName(String message, String title) { - while (true) { - String moduleName = JOptionPane.showInputDialog(SwingUtilities.getWindowAncestor(this), - message, title, JOptionPane.PLAIN_MESSAGE); - if (moduleName == null) { - return null; - } - - if (moduleName.isBlank()) { - MessageUtils.showError(this, "Please enter a module name"); - } else if (pipelineModuleDefinitionOperations() - .pipelineModuleDefinition(moduleName) != null) { - MessageUtils.showError(this, - moduleName + " already exists; please enter a unique module name"); - } else { - return moduleName; - } - } - } - private void showEditDialog(PipelineModuleDefinition module) { EditModuleDialog dialog = new EditModuleDialog(SwingUtilities.getWindowAncestor(this), module); @@ -177,10 +74,6 @@ private void showEditDialog(PipelineModuleDefinition module) { } } - private PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations() { - return pipelineModuleDefinitionOperations; - } - private static class ModuleLibraryTableModel extends AbstractZiggyTableModel implements DatabaseModel { diff --git a/src/main/java/gov/nasa/ziggy/ui/parameters/EditParameterSetDialog.java b/src/main/java/gov/nasa/ziggy/ui/parameters/EditParameterSetDialog.java index 6b5eb7c..e042561 100644 --- a/src/main/java/gov/nasa/ziggy/ui/parameters/EditParameterSetDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/parameters/EditParameterSetDialog.java @@ -14,6 +14,7 @@ import java.util.Set; import java.util.concurrent.ExecutionException; +import javax.swing.BorderFactory; import javax.swing.GroupLayout; import javax.swing.JLabel; import javax.swing.JPanel; @@ -42,7 +43,6 @@ public class EditParameterSetDialog extends javax.swing.JDialog { private final ParameterSet parameterSet; private Set currentParams; private PropertySheetPanel paramsPropPanel; - private JTextArea descriptionTextArea; private boolean cancelled; private boolean isNew; @@ -80,8 +80,10 @@ private JPanel createDataPanel() { JLabel versionTextField = new JLabel(Integer.toString(parameterSet.getVersion())); JLabel description = boldLabel("Description"); - descriptionTextArea = new JTextArea(parameterSet.getDescription()); + JTextArea descriptionTextArea = new JTextArea(parameterSet.getDescription()); + descriptionTextArea.setEditable(false); JScrollPane descriptionScrollPane = new JScrollPane(descriptionTextArea); + descriptionScrollPane.setBorder(BorderFactory.createEmptyBorder()); JPanel buttonPanel = ZiggySwingUtils.createButtonPanel(ButtonPanelContext.TOOL_BAR, ZiggySwingUtils.createButton(RESTORE_DEFAULTS, this::defaults)); @@ -134,8 +136,7 @@ private void save(ActionEvent evt) { @Override protected Void doInBackground() throws Exception { paramsPropPanel.writeToObject(currentParams); - parametersOperations().updateParameterSet(parameterSet, currentParams, - descriptionTextArea.getText(), isNew); + parametersOperations().updateParameterSet(parameterSet, currentParams, isNew); return null; } diff --git a/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java b/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java index d2d6fea..98e1f3f 100644 --- a/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java @@ -9,12 +9,14 @@ import java.util.ArrayList; import java.util.Date; import java.util.List; -import java.util.Set; +import java.util.Map; import java.util.concurrent.ExecutionException; +import javax.swing.Action; import javax.swing.JButton; import javax.swing.JFileChooser; import javax.swing.JOptionPane; +import javax.swing.ListSelectionModel; import javax.swing.SwingUtilities; import javax.swing.SwingWorker; import javax.swing.tree.DefaultMutableTreeNode; @@ -54,11 +56,11 @@ public class ViewEditParameterSetsPanel extends AbstractViewEditGroupPanel treeModel) { @@ -69,6 +71,10 @@ public ViewEditParameterSetsPanel(RowModel rowModel, ZiggyTreeModel treeModel = new ZiggyTreeModel<>(ParameterSet.class, + ZiggyTreeModel treeModel = new ZiggyTreeModel<>(TYPE, () -> new ParametersOperations().parameterSets()); ParameterSetsRowModel rowModel = new ParameterSetsRowModel(); return new ViewEditParameterSetsPanel(rowModel, treeModel); @@ -98,6 +104,12 @@ protected List buttons() { return buttons; } + @Override + protected void updateActionState(Map actionByFunction) { + actionByFunction.get(OptionalViewEditFunction.EDIT) + .setEnabled(ziggyTable.getTable().getSelectedRowCount() == 1); + } + private void report(ActionEvent evt) { Object[] options = { "Formatted", "Colon-delimited" }; @@ -208,12 +220,6 @@ protected void done() { }.execute(); } - @Override - protected Set optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.COPY, - OptionalViewEditFunction.RENAME); - } - @Override public void refresh() { try { @@ -223,11 +229,6 @@ public void refresh() { } } - @Override - protected void create() { - throw new UnsupportedOperationException("Create not supported"); - } - @Override protected void edit(int row) { try { @@ -244,123 +245,14 @@ protected void edit(int row) { } @Override - protected void copy(int row) { - - String newParameterSetName = readParameterSetName( - "Enter the name for the new parameter set", "New parameter set"); - if (newParameterSetName == null) { - return; - } - - ParameterSet newParameterSet = ziggyTable.getContentAtViewRow(row).newInstance(); - newParameterSet.setName(newParameterSetName); - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - parametersOperations().save(newParameterSet); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditParameterSetsPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void rename(int row) { - - String newParameterSetName = readParameterSetName( - "Enter the new name for this parameter set", "Rename parameter set"); - if (newParameterSetName == null) { - return; - } - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - parametersOperations().rename(ziggyTable.getContentAtViewRow(row), - newParameterSetName); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditParameterSetsPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void delete(int row) { - - ParameterSet selectedParameterSet = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete parameter set '" + selectedParameterSet.getName() - + "'?"); - if (choice != JOptionPane.YES_OPTION) { - return; - } - - new SwingWorker() { - - @Override - protected Void doInBackground() throws Exception { - parametersOperations().delete(selectedParameterSet); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditParameterSetsPanel.this, e); - } - } - }.execute(); - } - - private String readParameterSetName(String message, String title) { - while (true) { - String parameterSetName = JOptionPane.showInputDialog( - SwingUtilities.getWindowAncestor(this), message, title, JOptionPane.PLAIN_MESSAGE); - if (parameterSetName == null) { - return null; - } - if (parameterSetName.isBlank()) { - MessageUtils.showError(this, "Please enter a parameter set name"); - } else if (parametersOperations().parameterSet(parameterSetName) != null) { - MessageUtils.showError(this, - parameterSetName + " already exists; please enter a unique parameter set name"); - } else { - return parameterSetName; - } - } + protected String getType() { + return TYPE; } private ParameterImportExportOperations parameterImportExportOperations() { return parameterImportExportOperations; } - private ParametersOperations parametersOperations() { - return parametersOperations; - } - /** * Implementation of {@link RowModel} for a NetBeans {@link Outline} of {@link ParameterSet} * instances. diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java deleted file mode 100644 index 73b2c16..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java +++ /dev/null @@ -1,284 +0,0 @@ -package gov.nasa.ziggy.ui.pipeline; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.SAVE; - -import java.awt.BorderLayout; -import java.awt.FlowLayout; -import java.awt.GridBagConstraints; -import java.awt.GridBagLayout; -import java.awt.Insets; -import java.awt.Window; -import java.awt.event.ActionEvent; -import java.util.List; - -import javax.swing.BorderFactory; -import javax.swing.DefaultComboBoxModel; -import javax.swing.JButton; -import javax.swing.JComboBox; -import javax.swing.JLabel; -import javax.swing.JPanel; -import javax.swing.JTextArea; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; -import gov.nasa.ziggy.ui.util.MessageUtils; -import gov.nasa.ziggy.uow.UnitOfWorkGenerator; - -/** - * Allows editing a new or existing {@link PipelineDefinitionNode}. - * - * @author Todd Klaus - */ -@SuppressWarnings("serial") -public class EditPipelineNodeDialog extends javax.swing.JDialog { - @SuppressWarnings("unused") - private static final Logger log = LoggerFactory.getLogger(EditPipelineNodeDialog.class); - private JLabel moduleLabel; - - private JPanel dataPanel; - private JTextArea errorTextArea; - private JLabel uowTypeLabel; - private JPanel uowPanel; - private JPanel buttonPanel; - private JComboBox moduleComboBox; - private JButton cancelButton; - private JButton saveButton; - private boolean savePressed = false; - - private JLabel uowFullNameLabel; - - private final PipelineDefinition pipeline; - private final PipelineDefinitionNode pipelineNode; - - private final PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations = new PipelineModuleDefinitionOperations(); - - public EditPipelineNodeDialog(Window owner, PipelineDefinition pipeline, - PipelineDefinitionNode pipelineNode) throws Exception { - - super(owner, DEFAULT_MODALITY_TYPE); - this.pipeline = pipeline; - this.pipelineNode = pipelineNode; - - buildComponent(); - setLocationRelativeTo(owner); - } - - private void save(ActionEvent evt) { - - try { - PipelineModuleDefinition selectedModule = (PipelineModuleDefinition) moduleComboBox - .getSelectedItem(); - pipelineNode.setPipelineModuleDefinition(selectedModule); - - setVisible(false); - } catch (Exception e) { - MessageUtils.showError(this, e); - } - - savePressed = true; - } - - private void cancel(ActionEvent evt) { - setVisible(false); - } - - private void buildComponent() throws Exception { - - setTitle("Edit pipeline node"); - - getContentPane().add(createDataPanel(), BorderLayout.CENTER); - getContentPane().add(getButtonPanel(), BorderLayout.SOUTH); - - pack(); - } - - private JComboBox getModuleComboBox() { - - if (moduleComboBox == null) { - String moduleName = pipelineNode.getModuleName(); - String currentModuleName = null; - - if (moduleName != null) { - currentModuleName = moduleName; - } - - int currentIndex = 0; - int initialIndex = 0; - DefaultComboBoxModel moduleComboBoxModel = new DefaultComboBoxModel<>(); - List modules = pipelineModuleDefinitionOperations() - .allPipelineModuleDefinitions(); - for (PipelineModuleDefinition module : modules) { - moduleComboBoxModel.addElement(module); - if (currentModuleName != null && module.getName().equals(currentModuleName)) { - initialIndex = currentIndex; - } - currentIndex++; - } - moduleComboBox = new JComboBox<>(); - moduleComboBox.setModel(moduleComboBoxModel); - moduleComboBox.setMaximumRowCount(15); - moduleComboBox.setSelectedIndex(initialIndex); - - moduleComboBox.setEnabled(!pipeline.isLocked()); - } - - return moduleComboBox; - } - - private JPanel createDataPanel() { - - if (dataPanel == null) { - dataPanel = new JPanel(); - GridBagLayout dataPanelLayout = new GridBagLayout(); - dataPanelLayout.columnWeights = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1 }; - dataPanelLayout.columnWidths = new int[] { 7, 7, 7, 7, 7, 7 }; - dataPanelLayout.rowWeights = new double[] { 0.1, 0.1, 0.1, 0.1 }; - dataPanelLayout.rowHeights = new int[] { 7, 7, 7, 7 }; - dataPanel.setLayout(dataPanelLayout); - dataPanel.add(getModuleComboBox(), - new GridBagConstraints(2, 0, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - dataPanel.add(getModuleLabel(), - new GridBagConstraints(1, 0, 1, 1, 0.0, 0.0, GridBagConstraints.LINE_START, - GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - dataPanel.add(getUowPanel(), new GridBagConstraints(1, 2, 4, 3, 0.0, 1.0, - GridBagConstraints.CENTER, GridBagConstraints.BOTH, new Insets(0, 0, 0, 0), 0, 0)); - dataPanel.add(getErrorTextArea(), new GridBagConstraints(1, 1, 4, 1, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.BOTH, new Insets(0, 0, 0, 0), 0, 0)); - } - - return dataPanel; - } - - private JPanel getButtonPanel() { - - if (buttonPanel == null) { - buttonPanel = new JPanel(); - FlowLayout buttonPanelLayout = new FlowLayout(); - buttonPanelLayout.setHgap(40); - buttonPanel.setLayout(buttonPanelLayout); - buttonPanel.add(getSaveButton()); - buttonPanel.add(getCancelButton()); - } - - return buttonPanel; - } - - private JButton getSaveButton() { - - if (saveButton == null) { - saveButton = new JButton(); - saveButton.setText(SAVE); - saveButton.addActionListener(this::save); - saveButton.setEnabled(!pipeline.isLocked()); - } - - return saveButton; - } - - private JButton getCancelButton() { - - if (cancelButton == null) { - cancelButton = new JButton(); - cancelButton.setText(CANCEL); - cancelButton.addActionListener(this::cancel); - } - - return cancelButton; - } - - private JTextArea getErrorTextArea() { - if (errorTextArea == null) { - errorTextArea = new JTextArea(); - errorTextArea.setEditable(false); - errorTextArea.setForeground(new java.awt.Color(255, 0, 0)); - errorTextArea.setOpaque(false); - errorTextArea.setLineWrap(true); - errorTextArea.setWrapStyleWord(true); - } - return errorTextArea; - } - - private JLabel getModuleLabel() { - if (moduleLabel == null) { - moduleLabel = new JLabel(); - moduleLabel.setText("Module"); - } - return moduleLabel; - } - - /** - * @return Returns the savePressed. - */ - public boolean wasSavePressed() { - return savePressed; - } - - private JPanel getUowPanel() { - if (uowPanel == null) { - uowPanel = new JPanel(); - GridBagLayout uowPanelLayout = new GridBagLayout(); - uowPanelLayout.rowWeights = new double[] { 0.1, 0.1, 0.1, 0.1 }; - uowPanelLayout.rowHeights = new int[] { 7, 7, 7, 7 }; - uowPanelLayout.columnWeights = new double[] { 0.1, 0.1, 0.1, 0.1 }; - uowPanelLayout.columnWidths = new int[] { 7, 7, 7, 7 }; - uowPanel.setLayout(uowPanelLayout); - uowPanel.setBorder(BorderFactory.createTitledBorder("Unit of Work")); - uowPanel.add(getUowTypeLabel(), - new GridBagConstraints(0, 1, 1, 1, 0.0, 0.0, GridBagConstraints.LINE_START, - GridBagConstraints.NONE, new Insets(0, 0, 0, 0), 0, 0)); - uowPanel.add(getUowFullNameLabel(), - new GridBagConstraints(0, 3, 4, 1, 0.0, 0.0, GridBagConstraints.LINE_START, - GridBagConstraints.HORIZONTAL, new Insets(0, 0, 0, 0), 0, 0)); - } - return uowPanel; - } - - private JLabel getUowTypeLabel() { - if (uowTypeLabel == null) { - uowTypeLabel = new JLabel(); - ClassWrapper uowWrapper = pipelineModuleDefinitionOperations() - .unitOfWorkGenerator(pipelineNode.getModuleName()); - String uowName = uowWrapper.getClassName(); - String uowLabel = "Unit of Work Class: " + uowName; - uowTypeLabel.setText(uowLabel); - } - return uowTypeLabel; - } - - /** - * If the combination of module/uowtype/checkbox is in an invalid state, display error text and - * disable the save button until it's fixed - * - * @param message - */ - @SuppressWarnings("unused") - private void setError(String message) { - if (saveButton != null) { - saveButton.setEnabled(message.isBlank()); - } - - if (errorTextArea != null) { - errorTextArea.setText(message); - } - } - - private JLabel getUowFullNameLabel() { - if (uowFullNameLabel == null) { - uowFullNameLabel = new JLabel(); - uowFullNameLabel.setFont(new java.awt.Font("Dialog", 2, 10)); - } - return uowFullNameLabel; - } - - private PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations() { - return pipelineModuleDefinitionOperations; - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/NewPipelineDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/NewPipelineDialog.java deleted file mode 100644 index 3adedd8..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/NewPipelineDialog.java +++ /dev/null @@ -1,94 +0,0 @@ -package gov.nasa.ziggy.ui.pipeline; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CREATE; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.boldLabel; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; - -import java.awt.BorderLayout; -import java.awt.Window; -import java.awt.event.ActionEvent; - -import javax.swing.GroupLayout; -import javax.swing.JLabel; -import javax.swing.JPanel; -import javax.swing.JTextField; -import javax.swing.LayoutStyle.ComponentPlacement; - -import gov.nasa.ziggy.ui.ZiggyGuiConstants; -import gov.nasa.ziggy.ui.util.ZiggySwingUtils; - -/** - * @author Todd Klaus - * @author Bill Wohler - */ -@SuppressWarnings("serial") -public class NewPipelineDialog extends javax.swing.JDialog { - private JTextField nameTextField; - - private boolean cancelled; - - public NewPipelineDialog(Window owner) { - super(owner, DEFAULT_MODALITY_TYPE); - buildComponent(); - setLocationRelativeTo(owner); - } - - private void buildComponent() { - setTitle("New pipeline definition"); - - getContentPane().add(createDataPanel(), BorderLayout.CENTER); - getContentPane().add(createButtonPanel(createButton(CREATE, this::create), - createButton(CANCEL, this::cancel)), BorderLayout.SOUTH); - setPreferredSize(ZiggyGuiConstants.MIN_DIALOG_SIZE); - - pack(); - } - - private JPanel createDataPanel() { - JLabel name = boldLabel("Pipeline name"); - nameTextField = new JTextField(); - - JPanel dataPanel = new JPanel(); - GroupLayout dataPanelLayout = new GroupLayout(dataPanel); - dataPanelLayout.setAutoCreateContainerGaps(true); - dataPanel.setLayout(dataPanelLayout); - - dataPanelLayout.setHorizontalGroup( - dataPanelLayout.createParallelGroup().addComponent(name).addComponent(nameTextField)); - - dataPanelLayout.setVerticalGroup(dataPanelLayout.createSequentialGroup() - .addComponent(name) - .addComponent(nameTextField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, - GroupLayout.PREFERRED_SIZE) - .addPreferredGap(ComponentPlacement.UNRELATED)); - - return dataPanel; - } - - private void create(ActionEvent evt) { - setVisible(false); - } - - private void cancel(ActionEvent evt) { - setCancelled(true); - setVisible(false); - } - - public String getPipelineName() { - return nameTextField.getText(); - } - - public boolean isCancelled() { - return cancelled; - } - - public void setCancelled(boolean cancelled) { - this.cancelled = cancelled; - } - - public static void main(String[] args) { - ZiggySwingUtils.displayTestDialog(new NewPipelineDialog(null)); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java index 9c2f101..975832f 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java @@ -1,9 +1,7 @@ package gov.nasa.ziggy.ui.pipeline; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DELETE; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DIALOG; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EDIT; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.NEW; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createMenuItem; @@ -79,11 +77,9 @@ public ParameterSetMapEditorPanel(Map moduleParameterSetBy private void buildComponent() { JPanel toolBar = ZiggySwingUtils.createButtonPanel(ButtonPanelContext.TOOL_BAR, - createButton(NEW, "Add a new parameter set.", this::add), createButton(EDIT, "Edit this parameter set.", this::editParamValues)); - contextMenu = ZiggySwingUtils.createPopupMenu(createMenuItem(EDIT + DIALOG, this::edit), - createMenuItem(DELETE, this::delete)); + contextMenu = ZiggySwingUtils.createPopupMenu(createMenuItem(EDIT + DIALOG, this::edit)); paramSetMapTableModel = new ParameterSetNamesTableModel(moduleParameterSetByName, pipelineParameterSetByName); ziggyTable = new ZiggyTable<>(paramSetMapTableModel); @@ -123,28 +119,6 @@ public void mousePressed(java.awt.event.MouseEvent e) { }); } - private void add(ActionEvent evt) { - - ParameterSet newParameterSet = ParameterSetSelectorDialog - .selectParameterSet(SwingUtilities.getWindowAncestor(this)); - - if (newParameterSet != null) { - - if (moduleParameterSetByName.containsKey(newParameterSet.getName())) { - MessageUtils.showError(this, "A parameter set with name " + newParameterSet.getName() - + " already exists, use 'select' to change the existing instance"); - } else { - moduleParameterSetByName.put(newParameterSet.getName(), newParameterSet); - - if (mapListener != null) { - mapListener.notifyMapChanged(this); - } - - paramSetMapTableModel.update(moduleParameterSetByName, pipelineParameterSetByName); - } - } - } - private void editParamValues(ActionEvent evt) { int selectedRow = ziggyTable.getSelectedRow(); @@ -196,18 +170,6 @@ protected void done() { }.execute(); } - private void delete(ActionEvent evt) { - String parameterSetName = ziggyTable.getContentAtViewRow(selectedModelIndex) - .getAssignedName(); - moduleParameterSetByName.remove(parameterSetName); - - if (mapListener != null) { - mapListener.notifyMapChanged(this); - } - - paramSetMapTableModel.update(moduleParameterSetByName, pipelineParameterSetByName); - } - public ParameterSetMapEditorListener getMapListener() { return mapListener; } diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorDialog.java deleted file mode 100644 index 42f8d00..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorDialog.java +++ /dev/null @@ -1,71 +0,0 @@ -package gov.nasa.ziggy.ui.pipeline; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.SELECT; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; - -import java.awt.BorderLayout; -import java.awt.Window; -import java.awt.event.ActionEvent; - -import javax.swing.JDialog; - -import gov.nasa.ziggy.pipeline.definition.ParameterSet; - -/** - * {@link JDialog} wrapper for the {@link ParameterSetSelectorPanel} - * - * @author Todd Klaus - * @author Bill Wohler - */ -@SuppressWarnings("serial") -public class ParameterSetSelectorDialog extends javax.swing.JDialog { - - private ParameterSetSelectorPanel parameterSetSelectorPanel; - private boolean cancelled = false; - - private ParameterSetSelectorDialog(Window owner) { - super(owner, DEFAULT_MODALITY_TYPE); - buildComponent(); - setLocationRelativeTo(owner); - } - - private void buildComponent() { - setTitle("Select parameter set"); - - getContentPane().add(createDataPanel(), BorderLayout.CENTER); - getContentPane().add(createButtonPanel(createButton(SELECT, this::select), - createButton(CANCEL, this::cancel)), BorderLayout.SOUTH); - - pack(); - } - - private ParameterSetSelectorPanel createDataPanel() { - parameterSetSelectorPanel = new ParameterSetSelectorPanel(); - - return parameterSetSelectorPanel; - } - - private void select(ActionEvent evt) { - setVisible(false); - } - - private void cancel(ActionEvent evt) { - cancelled = true; - setVisible(false); - } - - /** - * Select a parameter set from the parameter set library with no filtering. - */ - public static ParameterSet selectParameterSet(Window owner) { - ParameterSetSelectorDialog dialog = new ParameterSetSelectorDialog(owner); - dialog.setVisible(true); - - if (dialog.cancelled) { - return null; - } - return dialog.parameterSetSelectorPanel.getSelected(); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorPanel.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorPanel.java deleted file mode 100644 index 05452e6..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetSelectorPanel.java +++ /dev/null @@ -1,80 +0,0 @@ -package gov.nasa.ziggy.ui.pipeline; - -import java.awt.BorderLayout; -import java.awt.Dimension; -import java.util.LinkedList; -import java.util.List; - -import javax.swing.AbstractListModel; -import javax.swing.JList; -import javax.swing.JScrollPane; - -import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.pipeline.definition.database.ParametersOperations; -import gov.nasa.ziggy.ui.util.models.DatabaseModel; - -/** - * Select a {@link ParameterSet} from a list or create a new one. - * - * @author Todd Klaus - * @author Bill Wohler - */ -@SuppressWarnings("serial") -public class ParameterSetSelectorPanel extends javax.swing.JPanel { - private JList paramSetList; - private ParameterSetListModel paramSetListModel; - - public ParameterSetSelectorPanel() { - buildComponent(); - } - - private void buildComponent() { - setLayout(new BorderLayout()); - setPreferredSize(new Dimension(400, 300)); - - paramSetListModel = new ParameterSetListModel(); - paramSetList = new JList<>(paramSetListModel); - add(new JScrollPane(paramSetList), BorderLayout.CENTER); - } - - public ParameterSet getSelected() { - int selectedIndex = paramSetList.getSelectedIndex(); - - if (selectedIndex != -1) { - return paramSetListModel.getElementAt(selectedIndex); - } - return null; - } - - private static class ParameterSetListModel extends AbstractListModel - implements DatabaseModel { - - private List paramSets = new LinkedList<>(); - - private final ParametersOperations parametersOperations = new ParametersOperations(); - - public ParameterSetListModel() { - loadFromDatabase(); - } - - @Override - public void loadFromDatabase() { - paramSets = parametersOperations().parameterSets(); - fireContentsChanged(this, 0, paramSets.size() - 1); - } - - @Override - public ParameterSet getElementAt(int index) { - return paramSets.get(index); - } - - @Override - public int getSize() { - return paramSets.size(); - } - - private ParametersOperations parametersOperations() { - return parametersOperations; - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java index 37fec49..cc4962b 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java @@ -217,7 +217,7 @@ private JButton createCloseButton() { JButton closeButton = ZiggySwingUtils.createButton(CLOSE, htmlBuilder("Close this dialog box.").appendBreak() .append( - "Your worker resource parameter changes won't be saved until you hit Save on the edit pipeline dialog.") + "Your worker resource parameter changes won't be saved until you press the Save button on the edit pipeline dialog.") .toString(), this::updateWorkerResources); @@ -226,7 +226,7 @@ private JButton createCloseButton() { // focus, the resulting event causes the validity of that box to become false for // a blink, which causes the close button to become disabled for a // blink; as a result, the buttons are disabled when the action listener would tell - // them to take their actions, and the user has to hit the button again to get it to + // them to take their actions, and the user has to press the button again to get it to // take its action. By performing the action when the button gains focus, we get // back to needing only one button push to get the focus and perform the action. closeButton.addFocusListener(new FocusAdapter() { diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineGraphCanvas.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineGraphCanvas.java index 0c95866..75e68f7 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineGraphCanvas.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineGraphCanvas.java @@ -2,7 +2,6 @@ import java.awt.BasicStroke; import java.awt.BorderLayout; -import java.awt.Component; import java.awt.Dimension; import java.awt.Graphics; import java.awt.Graphics2D; @@ -10,73 +9,44 @@ import java.awt.Rectangle; import java.awt.RenderingHints; import java.awt.Stroke; -import java.awt.event.ActionEvent; -import java.awt.event.MouseAdapter; -import java.awt.event.MouseEvent; -import java.util.ArrayList; -import java.util.List; - -import javax.swing.JMenuItem; -import javax.swing.JOptionPane; +import java.awt.Toolkit; + import javax.swing.JPanel; -import javax.swing.JPopupMenu; import javax.swing.JScrollPane; -import javax.swing.SwingUtilities; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionNodeOperations; -import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; -import gov.nasa.ziggy.ui.util.MessageUtils; /** - * Display the pipeline nodes as a directed graph with pop-up menus on each node. + * Display the pipeline nodes as a directed graph. * * @author Todd Klaus + * @author Bill Wohler */ @SuppressWarnings("serial") public class PipelineGraphCanvas extends JPanel { - private static final Logger log = LoggerFactory.getLogger(PipelineGraphCanvas.class); - private enum DrawType { LINES, NODES } - // size of the canvas (TODO: make dynamic based on # nodes) - private static final int CANVAS_HEIGHT = 750; - private static final int CANVAS_WIDTH = 500; - - // spacing at the top and left of the canvas + // Spacing at the top and left of the canvas. private final static int HORIZONTAL_INSET = 20; private final static int VERTICAL_INSET = 20; - // spacing between rows & columns of widgets + // Spacing between rows & columns of widgets. private final static int ROW_SPACING = 100; private final static int COLUMN_SPACING = 220; - // widget sizes + // Widget sizes. private final static int START_WIDGET_WIDTH = 100; private final static int START_WIDGET_HEIGHT = 30; private final static int NODE_WIDGET_WIDTH = 250; private final static int NODE_WIDGET_HEIGHT = 30; private final PipelineDefinition pipeline; - private PipelineNodeWidget selectedNodeWidget; - - private JPopupMenu canvasPopupMenu; private JPanel canvasPane; - private JScrollPane scrollPane; - private JMenuItem insertBeforeMenuItem; - private JMenuItem insertAfterMenuItem; - private JMenuItem insertBranchMenuItem; - protected JMenuItem editMenuItem; - protected JMenuItem deleteMenuItem; - - private final PipelineDefinitionOperations pipelineDefinitionOperations = new PipelineDefinitionOperations(); - private final PipelineDefinitionNodeOperations pipelineDefinitionNodeOperations = new PipelineDefinitionNodeOperations(); + private int width; + private int height = 2 * VERTICAL_INSET; public PipelineGraphCanvas(PipelineDefinition pipeline) { this.pipeline = pipeline; @@ -84,19 +54,31 @@ public PipelineGraphCanvas(PipelineDefinition pipeline) { } private void buildComponent() { - BorderLayout layout = new BorderLayout(); - setLayout(layout); - scrollPane = getScrollPane(); - add(scrollPane, BorderLayout.CENTER); - - // TODO Fix or delete menu - // Commands currently throw NullPointerException, so hide for now. - // setComponentPopupMenu(canvasPane, getCanvasPopupMenu()); + setLayout(new BorderLayout()); + createCanvasPane(); + add(new JScrollPane(canvasPane), BorderLayout.CENTER); } - public void redraw() { + private void createCanvasPane() { + canvasPane = new JPanel() { + @Override + protected void paintComponent(Graphics g) { + super.paintComponent(g); + // TODO Currently only supports one root node + drawPipeline(pipeline, DrawType.LINES, (Graphics2D) g); + } + }; + canvasPane.setBackground(new java.awt.Color(255, 255, 255)); + canvasPane.setLayout(null); + redraw(); + } + + private void redraw() { canvasPane.removeAll(); drawPipeline(pipeline, DrawType.NODES, null); + Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize(); + canvasPane.setPreferredSize( + new Dimension(Math.min(width, screenSize.width), Math.min(height, screenSize.height))); revalidate(); repaint(); } @@ -107,9 +89,9 @@ private void drawPipeline(PipelineDefinition pipeline, DrawType drawType, Graphi START_WIDGET_HEIGHT); if (drawType == DrawType.NODES) { - PipelineNodeWidget widget = new PipelineNodeWidget(pipeline); + PipelineNodeWidget widget = new PipelineNodeWidget(); canvasPane.add(widget); - widget.setBounds(nodeBounds); + adjustBounds(nodeBounds, widget, START_WIDGET_HEIGHT); } int childColumn = 0; @@ -125,9 +107,9 @@ private void drawNodes(PipelineDefinitionNode node, PipelineDefinitionNode paren VERTICAL_INSET + ROW_SPACING * row, NODE_WIDGET_WIDTH, NODE_WIDGET_HEIGHT); if (drawType == DrawType.NODES) { - PipelineNodeWidget widget = new PipelineNodeWidget(node, parentNode); + PipelineNodeWidget widget = new PipelineNodeWidget(node); canvasPane.add(widget); - widget.setBounds(nodeBounds); + adjustBounds(nodeBounds, widget, ROW_SPACING); } else if (parentBounds != null) { drawArrow(g2, parentBounds, nodeBounds); } @@ -140,6 +122,15 @@ private void drawNodes(PipelineDefinitionNode node, PipelineDefinitionNode paren } } + private void adjustBounds(Rectangle nodeBounds, PipelineNodeWidget widget, int widgetHeight) { + int widgetWidth = widget.getPreferredSize().width + 2 * HORIZONTAL_INSET; + nodeBounds.width = widgetWidth; + widget.setBounds(nodeBounds); + // TODO Increase width if there are multiple children + width = Math.max(width, widgetWidth + 2 * HORIZONTAL_INSET); + height += widgetHeight; + } + private void drawArrow(Graphics2D g2, Rectangle start, Rectangle end) { Stroke oldStroke = g2.getStroke(); @@ -181,317 +172,4 @@ private void drawArrow(Graphics2D g2, Rectangle start, Rectangle end) { g2.setStroke(oldStroke); } - - private void showEditDialog(PipelineDefinitionNode pipelineNode) { - - try { - EditPipelineNodeDialog dialog = new EditPipelineNodeDialog( - SwingUtilities.getWindowAncestor(this), pipeline, pipelineNode); - dialog.setVisible(true); - - if (dialog.wasSavePressed()) { - pipelineDefinitionOperations().merge(pipeline); - redraw(); - } - } catch (Throwable e) { - MessageUtils.showError(this, e); - } - } - - private void enableMenuItemsPerContext(PipelineNodeWidget selectedNodeWidget) { - if (pipeline.isLocked()) { - insertAfterMenuItem.setEnabled(false); - insertBranchMenuItem.setEnabled(false); - insertBeforeMenuItem.setEnabled(false); - // editMenuItem.setEnabled(false); - deleteMenuItem.setEnabled(false); - } else if (selectedNodeWidget.isStartNode()) { - insertBeforeMenuItem.setEnabled(false); - editMenuItem.setEnabled(false); - deleteMenuItem.setEnabled(false); - } else { - insertBeforeMenuItem.setEnabled(true); - editMenuItem.setEnabled(true); - deleteMenuItem.setEnabled(true); - } - } - - private void edit(ActionEvent evt) { - showEditDialog(selectedNodeWidget.getPipelineNode()); - } - - private void delete(ActionEvent evt) { - try { - PipelineDefinitionNode selectedNode = selectedNodeWidget.getPipelineNode(); - PipelineDefinitionNode parentNode = selectedNodeWidget.getPipelineNodeParent(); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete pipelineNode '" + selectedNode.getId() + "'?"); - - if (choice == JOptionPane.YES_OPTION) { - List currentNextNodes = null; - - if (parentNode != null) { - currentNextNodes = parentNode.getNextNodes(); - } else { - currentNextNodes = pipeline.getRootNodes(); - } - - // remove selected node from parent's nextNodes list - currentNextNodes.remove(selectedNode); - - // add selected node's nextNodes to parent's nextNodes list - currentNextNodes.addAll(selectedNode.getNextNodes()); - - pipelineDefinitionNodeOperations().delete(selectedNode); - - redraw(); - } - } catch (Throwable e) { - MessageUtils.showError(this, e); - } - } - - private void insertBefore(ActionEvent evt) { - try { - PipelineDefinitionNode selectedNode = selectedNodeWidget.getPipelineNode(); - PipelineDefinitionNode predecessorNode = selectedNodeWidget.getPipelineNodeParent(); - PipelineDefinitionNode newNode = new PipelineDefinitionNode(); - - EditPipelineNodeDialog editPipelineNodeDialog = new EditPipelineNodeDialog( - SwingUtilities.getWindowAncestor(this), pipeline, newNode); - editPipelineNodeDialog.setVisible(true); - - if (editPipelineNodeDialog.wasSavePressed()) { - List predecessorsNextNodesList = null; - if (predecessorNode != null) { - predecessorsNextNodesList = predecessorNode.getNextNodes(); - } else { - // inserting the first node - predecessorsNextNodesList = pipeline.getRootNodes(); - } - - // change predecessor's nextNodes entry for this node to new - // node - int selectedNodeIndex = predecessorsNextNodesList.indexOf(selectedNode); - assert selectedNodeIndex != -1 : "PipelineGraphCanvas:doInsertBefore: predecessor node does not point to this node!"; - - predecessorsNextNodesList.set(selectedNodeIndex, newNode); - - // set new node's nextNodes to this node - newNode.getNextNodes().add(selectedNode); - - pipelineDefinitionOperations().merge(pipeline); - redraw(); - } - } catch (Throwable e) { - MessageUtils.showError(this, e); - } - } - - private void insertAfter(ActionEvent evt) { - try { - PipelineDefinitionNode selectedNode = selectedNodeWidget.getPipelineNode(); - PipelineDefinitionNode newNode = new PipelineDefinitionNode(); - - EditPipelineNodeDialog editPipelineNodeDialog = new EditPipelineNodeDialog( - SwingUtilities.getWindowAncestor(this), pipeline, newNode); - editPipelineNodeDialog.setVisible(true); - - if (editPipelineNodeDialog.wasSavePressed()) { - List currentNextNodes = null; - - if (selectedNodeWidget.isStartNode()) { - currentNextNodes = selectedNodeWidget.getPipeline().getRootNodes(); - } else { - currentNextNodes = selectedNode.getNextNodes(); - } - - // copy selected node's nextNodes to new node's nextNodes - List newNodeNextNodes = new ArrayList<>(currentNextNodes); - newNode.setNextNodes(newNodeNextNodes); - - // set selected node's nextNodes to new node - List selectedNodeNextNodes = currentNextNodes; - selectedNodeNextNodes.clear(); - selectedNodeNextNodes.add(newNode); - - pipelineDefinitionOperations().merge(pipeline); - redraw(); - } - } catch (Throwable e) { - MessageUtils.showError(this, e); - } - } - - private void insertBranch(ActionEvent evt) { - try { - PipelineDefinitionNode selectedNode = selectedNodeWidget.getPipelineNode(); - PipelineDefinitionNode newNode = new PipelineDefinitionNode(); - - EditPipelineNodeDialog editPipelineNodeDialog = new EditPipelineNodeDialog( - SwingUtilities.getWindowAncestor(this), pipeline, newNode); - editPipelineNodeDialog.setVisible(true); - - if (editPipelineNodeDialog.wasSavePressed()) { - List currentNextNodes = null; - - if (selectedNodeWidget.isStartNode()) { - currentNextNodes = selectedNodeWidget.getPipeline().getRootNodes(); - } else { - currentNextNodes = selectedNode.getNextNodes(); - } - - // add new node to selected node's nextNodes - currentNextNodes.add(newNode); - - pipelineDefinitionOperations().merge(pipeline); - redraw(); - } - } catch (Throwable e) { - MessageUtils.showError(this, e); - } - } - - private void canvasPaneMouseClicked(MouseEvent evt) { - if (evt.getClickCount() == 2) { - Component selectedComponent = canvasPane.getComponentAt(evt.getX(), evt.getY()); - log.debug("MP:selectedComponent = " + selectedComponent); - if (selectedComponent instanceof PipelineNodeWidget) { - selectedNodeWidget = (PipelineNodeWidget) selectedComponent; - - // TODO Fix or delete command - // Command currently throws NullPointerException, so suppress for now. - // doEdit(); - } - } - } - - private JScrollPane getScrollPane() { - if (scrollPane == null) { - scrollPane = new JScrollPane(); - scrollPane.setViewportView(getCanvasPane()); - } - - return scrollPane; - } - - private JPanel getCanvasPane() { - if (canvasPane == null) { - canvasPane = new JPanel() { - @Override - protected void paintComponent(Graphics g) { - super.paintComponent(g); - // TODO: currently only supports one root node - drawPipeline(pipeline, DrawType.LINES, (Graphics2D) g); - } - }; - canvasPane.setBackground(new java.awt.Color(255, 255, 255)); - canvasPane.setLayout(null); - canvasPane.setPreferredSize(new Dimension(CANVAS_WIDTH, CANVAS_HEIGHT)); - canvasPane.addMouseListener(new MouseAdapter() { - @Override - public void mouseClicked(MouseEvent evt) { - canvasPaneMouseClicked(evt); - } - }); - redraw(); - } - - return canvasPane; - } - - @SuppressWarnings("unused") - private JPopupMenu getCanvasPopupMenu() { - if (canvasPopupMenu == null) { - canvasPopupMenu = new JPopupMenu(); - canvasPopupMenu.add(getInsertBeforeMenuItem()); - canvasPopupMenu.add(getInsertAfterMenuItem()); - canvasPopupMenu.add(getInsertBranchMenuItem()); - canvasPopupMenu.add(getEditMenuItem()); - canvasPopupMenu.add(getDeleteMenuItem()); - } - - return canvasPopupMenu; - } - - /** - * Auto-generated method for setting the popup menu for a component - */ - @SuppressWarnings("unused") - private void setComponentPopupMenu(final java.awt.Component parent, - final javax.swing.JPopupMenu menu) { - - parent.addMouseListener(new java.awt.event.MouseAdapter() { - @Override - public void mousePressed(java.awt.event.MouseEvent e) { - if (e.isPopupTrigger()) { - Component selectedComponent = canvasPane.getComponentAt(e.getX(), e.getY()); - if (selectedComponent instanceof PipelineNodeWidget) { - selectedNodeWidget = (PipelineNodeWidget) selectedComponent; - enableMenuItemsPerContext(selectedNodeWidget); - menu.show(parent, e.getX(), e.getY()); - } - } - } - }); - } - - private JMenuItem getEditMenuItem() { - if (editMenuItem == null) { - editMenuItem = new JMenuItem(); - editMenuItem.setText("View/Edit Node..."); - editMenuItem.addActionListener(this::edit); - } - - return editMenuItem; - } - - private JMenuItem getDeleteMenuItem() { - if (deleteMenuItem == null) { - deleteMenuItem = new JMenuItem(); - deleteMenuItem.setText("Delete Node..."); - deleteMenuItem.addActionListener(this::delete); - } - - return deleteMenuItem; - } - - private JMenuItem getInsertBranchMenuItem() { - if (insertBranchMenuItem == null) { - insertBranchMenuItem = new JMenuItem(); - insertBranchMenuItem.setText("Insert branch..."); - insertBranchMenuItem.addActionListener(this::insertBranch); - } - - return insertBranchMenuItem; - } - - private JMenuItem getInsertBeforeMenuItem() { - if (insertBeforeMenuItem == null) { - insertBeforeMenuItem = new JMenuItem(); - insertBeforeMenuItem.setText("Insert before..."); - insertBeforeMenuItem.addActionListener(this::insertBefore); - } - - return insertBeforeMenuItem; - } - - private JMenuItem getInsertAfterMenuItem() { - if (insertAfterMenuItem == null) { - insertAfterMenuItem = new JMenuItem(); - insertAfterMenuItem.setText("Insert after..."); - insertAfterMenuItem.addActionListener(this::insertAfter); - } - - return insertAfterMenuItem; - } - - private PipelineDefinitionOperations pipelineDefinitionOperations() { - return pipelineDefinitionOperations; - } - - private PipelineDefinitionNodeOperations pipelineDefinitionNodeOperations() { - return pipelineDefinitionNodeOperations; - } } diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java index 8a4fd7b..57a73dc 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java @@ -1,15 +1,11 @@ package gov.nasa.ziggy.ui.pipeline; import java.awt.Font; -import java.awt.GridBagConstraints; -import java.awt.GridBagLayout; -import java.awt.Insets; import javax.swing.BorderFactory; import javax.swing.JLabel; import javax.swing.border.BevelBorder; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.database.PipelineModuleDefinitionOperations; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; @@ -20,78 +16,45 @@ @SuppressWarnings("serial") public class PipelineNodeWidget extends javax.swing.JPanel { private JLabel label; - private PipelineDefinition pipeline = null; - private PipelineDefinitionNode pipelineNode = null; - private PipelineDefinitionNode pipelineNodeParent = null; private final PipelineModuleDefinitionOperations pipelineModuleDefinitionOperations = new PipelineModuleDefinitionOperations(); public PipelineNodeWidget() { - buildComponent(); + this(null); } - public PipelineNodeWidget(PipelineDefinitionNode pipelineNode, - PipelineDefinitionNode pipelineNodeParent) { - this.pipelineNode = pipelineNode; - this.pipelineNodeParent = pipelineNodeParent; - buildComponent(); + public PipelineNodeWidget(PipelineDefinitionNode pipelineNode) { + buildComponent(pipelineNode); } - public PipelineNodeWidget(PipelineDefinition pipeline) { - this.pipeline = pipeline; - buildComponent(); + public String text() { + return label.getText(); } - private void buildComponent() { - JLabel nodeLabel = getLabel(); - - setLayout(new GridBagLayout()); - setPreferredSize(nodeLabel.getPreferredSize()); + private void buildComponent(PipelineDefinitionNode pipelineNode) { + label = createLabel(pipelineNode); + setPreferredSize(label.getPreferredSize()); setBorder(BorderFactory.createEtchedBorder(BevelBorder.LOWERED)); - add(nodeLabel, new GridBagConstraints(0, 0, 1, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.NONE, new Insets(0, 0, 0, 0), 0, 0)); - } - - public boolean isStartNode() { - return pipeline != null; - } - - public PipelineDefinition getPipeline() { - return pipeline; - } - - /** - * @return Returns the pipelineNode. - */ - public PipelineDefinitionNode getPipelineNode() { - return pipelineNode; - } - - /** - * @return Returns the pipelineNodeParent. - */ - public PipelineDefinitionNode getPipelineNodeParent() { - return pipelineNodeParent; + add(label); } - private JLabel getLabel() { - if (label == null) { - label = new JLabel(); - if (pipelineNode == null) { - // START node - label.setText("START"); - label.setFont(new java.awt.Font("Dialog", Font.BOLD, 16)); - } else { - String uowtgShortName = "-"; - try { - uowtgShortName = pipelineModuleDefinitionOperations() - .unitOfWorkGenerator(pipelineNode.getModuleName()) - .newInstance() - .toString(); - } catch (Exception e) { - } - label.setText(pipelineNode.getModuleName() + " (" + uowtgShortName + ")"); + private JLabel createLabel(PipelineDefinitionNode pipelineNode) { + JLabel label = new JLabel(); + if (pipelineNode == null) { + // START node + label.setText("START"); + label.setFont(new java.awt.Font("Dialog", Font.BOLD, 16)); + } else { + String uowGeneratorName = "-"; + try { + uowGeneratorName = pipelineModuleDefinitionOperations() + .unitOfWorkGenerator(pipelineNode.getModuleName()) + .newInstance() + .getClass() + .getSimpleName(); + } catch (Exception e) { } + label.setText(pipelineNode.getModuleName() + " (" + uowGeneratorName + ")"); } return label; } diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java index e5dcdf0..2c98cd4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java @@ -164,7 +164,7 @@ private void buildComponent() { getContentPane().add(createDataPanel(), BorderLayout.CENTER); closeButton = createButton(CLOSE, htmlBuilder("Close this dialog box.").appendBreak() .append( - "Your remote parameter changes won't be saved until you hit Save on the edit pipeline dialog.") + "Your remote parameter changes won't be saved until you press the Save button on the edit pipeline dialog.") .toString(), this::close); getContentPane().add( createButtonPanel(closeButton, createButton(CANCEL, @@ -994,16 +994,16 @@ private void populateCurrentParameters() { private int textToInt(ValidityTestingFormattedTextField field) { try { - return Integer.parseInt(field.getText()); - } catch (NumberFormatException e) { + return NumberFormat.getInstance().parse(field.getText()).intValue(); + } catch (ParseException e) { return 0; } } private double textToDouble(ValidityTestingFormattedTextField field) { try { - return Double.parseDouble(field.getText()); - } catch (NumberFormatException e) { + return NumberFormat.getInstance().parse(field.getText()).doubleValue(); + } catch (ParseException e) { return 0.0; } } diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/StartPipelineDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/StartPipelineDialog.java index 73d9c9e..89870da 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/StartPipelineDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/StartPipelineDialog.java @@ -256,8 +256,8 @@ private void start(ActionEvent evt) { startNode == null ? "pipeline start" : startNode.getModuleName(), endNode == null ? "pipeline end" : endNode.getModuleName()); - ZiggyMessenger - .publish(new StartPipelineRequest(pipeline.getName(), instanceNameTextField.getText(), + ZiggyMessenger.publish( + new StartPipelineRequest(pipeline.getName(), instanceNameTextField.getText(), startNode, endNode, repetitions, delayMinutes * 60)); setCursor(null); diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java index b65d956..c3d7166 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java @@ -11,18 +11,19 @@ import java.awt.event.ActionEvent; import java.util.Date; import java.util.List; +import java.util.Map; import java.util.Set; +import javax.swing.AbstractAction; +import javax.swing.Action; import javax.swing.JButton; import javax.swing.JDialog; -import javax.swing.JOptionPane; +import javax.swing.ListSelectionModel; import javax.swing.SwingUtilities; -import javax.swing.SwingWorker; import javax.swing.tree.DefaultMutableTreeNode; import org.netbeans.swing.outline.RowModel; -import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; @@ -42,15 +43,20 @@ */ public class ViewEditPipelinesPanel extends AbstractViewEditGroupPanel { - private static final long serialVersionUID = 20240614L; + private static final long serialVersionUID = 20241002L; + private static final String TYPE = "PipelineDefinition"; - private final PipelineDefinitionOperations pipelineDefinitionOperations = new PipelineDefinitionOperations(); + private Action startAction; public ViewEditPipelinesPanel(PipelineRowModel rowModel, ZiggyTreeModel treeModel) { super(rowModel, treeModel, "Name"); buildComponent(); + ziggyTable.getTable() + .getSelectionModel() + .setSelectionMode(ListSelectionModel.MULTIPLE_INTERVAL_SELECTION); + ZiggyMessenger.subscribe(InvalidateConsoleModelsMessage.class, this::invalidateModel); } @@ -59,8 +65,7 @@ public ViewEditPipelinesPanel(PipelineRowModel rowModel, * for parameter sets needs the tree model in its constructor. */ public static ViewEditPipelinesPanel newInstance() { - ZiggyTreeModel treeModel = new ZiggyTreeModel<>( - PipelineDefinition.class, + ZiggyTreeModel treeModel = new ZiggyTreeModel<>(TYPE, () -> new PipelineDefinitionOperations().allPipelineDefinitions()); return new ViewEditPipelinesPanel(new PipelineRowModel(), treeModel); } @@ -69,24 +74,26 @@ private void invalidateModel(InvalidateConsoleModelsMessage message) { ziggyTable.loadFromDatabase(); } + @SuppressWarnings("serial") @Override protected List buttons() { List buttons = super.buttons(); - buttons.add(createButton(START, this::start)); + startAction = new AbstractAction(START, null) { + @Override + public void actionPerformed(ActionEvent evt) { + start(); + } + }; + buttons.add(createButton(startAction)); return buttons; } - private void start(ActionEvent evt) { + private void start() { int tableRow = ziggyTable.getSelectedRow(); selectedModelRow = ziggyTable.convertRowIndexToModel(tableRow); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(selectedModelRow); - // TODO Delete once the Start button is disabled if a non-pipeline row is selected - if (pipeline == null) { - return; - } - try { new StartPipelineDialog(SwingUtilities.getWindowAncestor(this), pipeline) .setVisible(true); @@ -96,38 +103,24 @@ private void start(ActionEvent evt) { } @Override - protected Set optionalViewEditFunctions() { - return Set.of( - /* TODO Implement OptionalViewEditFunctions.NEW, per ZIGGY-284 */ OptionalViewEditFunction.VIEW, - OptionalViewEditFunction.COPY, OptionalViewEditFunction.RENAME, - OptionalViewEditFunction.DELETE); + protected void updateActionState(Map actionByFunction) { + startAction.setEnabled(ziggyTable.getTable().getSelectedRowCount() == 1); + actionByFunction.get(OptionalViewEditFunction.EDIT) + .setEnabled(ziggyTable.getTable().getSelectedRowCount() == 1); + actionByFunction.get(OptionalViewEditFunction.VIEW) + .setEnabled(ziggyTable.getTable().getSelectedRowCount() == 1); } @Override - protected void refresh() { - try { - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(this, e); - } + protected Set optionalViewEditFunctions() { + return Set.of(OptionalViewEditFunction.VIEW); } @Override - protected void create() { - - NewPipelineDialog newPipelineDialog = new NewPipelineDialog( - SwingUtilities.getWindowAncestor(this)); - newPipelineDialog.setVisible(true); - if (newPipelineDialog.isCancelled()) { - return; - } - - new EditPipelineDialog(SwingUtilities.getWindowAncestor(this), - new PipelineDefinition(newPipelineDialog.getPipelineName())).setVisible(true); - + protected void refresh() { try { ziggyTable.loadFromDatabase(); - } catch (PipelineException e) { + } catch (Exception e) { MessageUtils.showError(this, e); } } @@ -172,115 +165,8 @@ protected void edit(int row) { } @Override - protected void copy(int row) { - - String newPipelineName = readPipelineName("Enter the name for the new pipeline definition", - "New pipeline definition"); - if (newPipelineName == null) { - return; - } - - PipelineDefinition newPipeline = ziggyTable.getContentAtViewRow(row).newInstance(); - newPipeline.setName(newPipelineName); - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - pipelineDefinitionOperations().merge(newPipeline); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditPipelinesPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void rename(int row) { - - String newPipelineName = readPipelineName("Enter the new name for this pipeline definition", - "Rename pipeline definition"); - if (newPipelineName == null) { - return; - } - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - pipelineDefinitionOperations().rename(ziggyTable.getContentAtViewRow(row), - newPipelineName); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditPipelinesPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void delete(int row) { - - PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete pipeline " + pipeline.getName() + "?"); - if (choice != JOptionPane.YES_OPTION) { - return; - } - - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - pipelineDefinitionOperations().delete(pipeline); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(ViewEditPipelinesPanel.this, e); - } - } - }.execute(); - } - - private String readPipelineName(String message, String title) { - while (true) { - String pipelineName = JOptionPane.showInputDialog( - SwingUtilities.getWindowAncestor(this), message, title, JOptionPane.PLAIN_MESSAGE); - if (pipelineName == null) { - return null; - } - if (pipelineName.isBlank()) { - MessageUtils.showError(this, "Please enter a pipeline name"); - } else if (pipelineDefinitionOperations().pipelineDefinition(pipelineName) != null) { - MessageUtils.showError(this, - pipelineName + " already exists; please enter a unique pipeline name"); - } else { - return pipelineName; - } - } - } - - private PipelineDefinitionOperations pipelineDefinitionOperations() { - return pipelineDefinitionOperations; + protected String getType() { + return TYPE; } private static class PipelineRowModel diff --git a/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java b/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java index 8a97b66..9df0717 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java @@ -6,13 +6,19 @@ import java.text.SimpleDateFormat; import java.util.LinkedList; import java.util.List; +import java.util.concurrent.ExecutionException; import javax.swing.GroupLayout; import javax.swing.JPanel; import javax.swing.JScrollPane; +import javax.swing.SwingWorker; import javax.swing.table.AbstractTableModel; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import gov.nasa.ziggy.services.alert.Alert; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.messages.AlertMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; @@ -28,7 +34,9 @@ * @author Bill Wohler */ public class AlertsStatusPanel extends JPanel { - private static final long serialVersionUID = 20230822L; + private static final long serialVersionUID = 20240924L; + + private static final Logger log = LoggerFactory.getLogger(AlertsStatusPanel.class); private AlertsTableModel alertsTableModel; @@ -70,7 +78,7 @@ private void acknowledge(ActionEvent evt) { private static class AlertsTableModel extends AbstractTableModel implements ModelContentClass { - private static final long serialVersionUID = 20230822L; + private static final long serialVersionUID = 20240924L; private static final int MAX_ALERTS = 1000; private static final String WARNING_MESSAGE = "WARN level alerts present"; @@ -95,23 +103,47 @@ public void clear() { } public void addAlertMessage(AlertMessage msg) { - if (alertMessages.size() >= MAX_ALERTS) { - // Remove oldest message from bottom. - alertMessages.remove(alertMessages.size() - 1); - } - - // Insert new message at top. - alertMessages.add(0, msg); - - fireTableRowsInserted(alertMessages.size() - 1, alertMessages.size() - 1); - - String severity = msg.getAlertData().getSeverity(); - Indicator alertIndicator = StatusPanel.ContentItem.ALERTS.menuItem(); - if (severity.equals("WARNING")) { - alertIndicator.setState(Indicator.State.WARNING, WARNING_MESSAGE); - } else if (severity.equals("ERROR") || severity.equals("INFRASTRUCTURE")) { - alertIndicator.setState(Indicator.State.ERROR, ERROR_MESSAGE); - } + new SwingWorker() { + + @Override + protected Void doInBackground() throws Exception { + if (alertMessages.size() >= MAX_ALERTS) { + // Remove oldest message from bottom. + alertMessages.remove(alertMessages.size() - 1); + } + + // Insert new message at top. + alertMessages.add(0, msg); + return null; + } + + @Override + protected void done() { + try { + get(); // check for exception + } catch (InterruptedException | ExecutionException e) { + log.error("Could not load pipeline module definitions", e); + } + + // Remember that we inserted the new message at the top. + fireTableRowsInserted(0, 0); + + Severity severity = msg.getAlertData().getSeverity(); + Indicator alertIndicator = StatusPanel.ContentItem.ALERTS.menuItem(); + + // If the state is already set to ERROR, it's already as bad as it'll get. + // In particular, don't change the state from ERROR to WARNING. + if (alertIndicator.getState() == Indicator.State.ERROR) { + return; + } + + if (severity == Severity.WARNING) { + alertIndicator.setState(Indicator.State.WARNING, WARNING_MESSAGE); + } else if (severity == Severity.ERROR || severity == Severity.INFRASTRUCTURE) { + alertIndicator.setState(Indicator.State.ERROR, ERROR_MESSAGE); + } + } + }.execute(); } @Override @@ -122,7 +154,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { case 0 -> formatter.format(alert.getTimestamp()); case 1 -> alert.getSourceComponent(); case 2 -> HostNameUtils.callerHostNameOrLocalhost(alert.getProcessHost()); - case 3 -> alert.getSourceTaskId(); + case 3 -> alert.getSourceTask(); case 4 -> alert.getSeverity(); case 5 -> alert.getMessage(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); diff --git a/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java b/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java index 511efa4..1e22f77 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java @@ -240,7 +240,7 @@ public synchronized Object getValueAt(int rowIndex, int columnIndex) { case 2 -> ZiggyStringUtils.elapsedTime(message.getProcessingStartTime(), System.currentTimeMillis()); case 3 -> message.getInstanceId(); - case 4 -> message.getTaskId(); + case 4 -> message.getPipelineTask(); case 5 -> message.getModule(); case 6 -> message.getModuleUow(); default -> ""; diff --git a/src/main/java/gov/nasa/ziggy/ui/util/EditKeyValuePairDialog.java b/src/main/java/gov/nasa/ziggy/ui/util/EditKeyValuePairDialog.java deleted file mode 100644 index 203d7ff..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/util/EditKeyValuePairDialog.java +++ /dev/null @@ -1,192 +0,0 @@ -package gov.nasa.ziggy.ui.util; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.SAVE; - -import java.awt.BorderLayout; -import java.awt.FlowLayout; -import java.awt.GridBagConstraints; -import java.awt.GridBagLayout; -import java.awt.Insets; -import java.awt.Window; -import java.awt.event.ActionEvent; -import java.util.concurrent.ExecutionException; - -import javax.swing.JButton; -import javax.swing.JLabel; -import javax.swing.JPanel; -import javax.swing.JTextField; -import javax.swing.SwingWorker; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.services.config.KeyValuePair; -import gov.nasa.ziggy.services.config.KeyValuePairOperations; - -@SuppressWarnings("serial") -public class EditKeyValuePairDialog extends javax.swing.JDialog { - private static final Logger log = LoggerFactory.getLogger(EditKeyValuePairDialog.class); - - private KeyValuePair keyValuePair; - private JPanel dataPanel; - private JPanel buttonPanel; - private JLabel keyLabel; - private JTextField valueText; - private JTextField keyText; - private JLabel valueLabel; - private JButton cancelButton; - private JButton saveButton; - - private final KeyValuePairOperations keyValuePairOperations = new KeyValuePairOperations(); - - public EditKeyValuePairDialog(Window owner, KeyValuePair keyValuePair) { - super(owner, "Edit Key/Value Pair"); - this.keyValuePair = keyValuePair; - buildComponent(); - setLocationRelativeTo(owner); - } - - private void buildComponent() { - try { - BorderLayout thisLayout = new BorderLayout(); - getContentPane().setLayout(thisLayout); - getContentPane().add(getDataPanel(), BorderLayout.CENTER); - getContentPane().add(getButtonPanel(), BorderLayout.SOUTH); - setSize(400, 200); - } catch (Exception e) { - log.error("buildComponent()", e); - } - } - - private JPanel getDataPanel() { - - if (dataPanel == null) { - dataPanel = new JPanel(); - GridBagLayout dataPanelLayout = new GridBagLayout(); - dataPanelLayout.columnWeights = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1 }; - dataPanelLayout.columnWidths = new int[] { 7, 7, 7, 7, 7, 7 }; - dataPanelLayout.rowWeights = new double[] { 0.1, 0.1 }; - dataPanelLayout.rowHeights = new int[] { 7, 7 }; - dataPanel.setLayout(dataPanelLayout); - dataPanel.add(getKeyLabel(), - new GridBagConstraints(0, 0, 1, 1, 0.0, 0.0, GridBagConstraints.LINE_END, - GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - dataPanel.add(getValueLabel(), - new GridBagConstraints(0, 1, 1, 1, 0.0, 0.0, GridBagConstraints.LINE_END, - GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - dataPanel.add(getKeyText(), - new GridBagConstraints(1, 0, 4, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - dataPanel.add(getValueText(), - new GridBagConstraints(1, 1, 4, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - } - - return dataPanel; - } - - private JPanel getButtonPanel() { - if (buttonPanel == null) { - buttonPanel = new JPanel(); - FlowLayout buttonPanelLayout = new FlowLayout(); - buttonPanelLayout.setHgap(40); - buttonPanel.setLayout(buttonPanelLayout); - buttonPanel.add(getSaveButton()); - buttonPanel.add(getCancelButton()); - } - - return buttonPanel; - } - - private JButton getSaveButton() { - if (saveButton == null) { - saveButton = new JButton(); - saveButton.setText(SAVE); - saveButton.addActionListener(this::save); - } - - return saveButton; - } - - private JButton getCancelButton() { - if (cancelButton == null) { - cancelButton = new JButton(); - cancelButton.setText(CANCEL); - cancelButton.addActionListener(this::cancel); - } - - return cancelButton; - } - - private JLabel getKeyLabel() { - if (keyLabel == null) { - keyLabel = new JLabel(); - keyLabel.setText("Key"); - } - - return keyLabel; - } - - private JLabel getValueLabel() { - if (valueLabel == null) { - valueLabel = new JLabel(); - valueLabel.setText("Value"); - } - - return valueLabel; - } - - private JTextField getKeyText() { - if (keyText == null) { - keyText = new JTextField(); - keyText.setEditable(false); - keyText.setText(keyValuePair.getKey()); - } - - return keyText; - } - - private JTextField getValueText() { - if (valueText == null) { - valueText = new JTextField(); - valueText.setText(keyValuePair.getValue()); - } - - return valueText; - } - - private void save(ActionEvent evt) { - keyValuePair.setValue(valueText.getText()); - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - keyValuePairOperations().persist(keyValuePair); - return null; - } - - @Override - protected void done() { - try { - get(); // check for exception - setVisible(false); - } catch (InterruptedException | ExecutionException e) { - MessageUtils.showError(getRootPane(), e); - } - } - }.execute(); - } - - private void cancel(ActionEvent evt) { - setVisible(false); - } - - private KeyValuePairOperations keyValuePairOperations() { - return keyValuePairOperations; - } - - public static void main(String[] args) { - ZiggySwingUtils - .displayTestDialog(new EditKeyValuePairDialog(null, new KeyValuePair("key", "value"))); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java b/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java index 868f8d0..b2a933f 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java @@ -11,7 +11,6 @@ import com.google.common.collect.Sets; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.pipeline.definition.database.GroupOperations; /** @@ -20,7 +19,7 @@ * @author PT * @author Bill Wohler */ -public class GroupInformation { +public class GroupInformation { private Map objectByName; private Map> objectsByGroup; @@ -33,8 +32,8 @@ public class GroupInformation { * Creates a {@code GroupInformation} object. This makes a database call so it should be * performed in {@link SwingWorker#doInBackground()}. */ - public GroupInformation(Class clazz, List allObjects) { - List allGroups = groupOperations().groupsForClass(clazz); + public GroupInformation(String type, List allObjects) { + List allGroups = groupOperations().groups(type); objectByName = createObjectByName(allObjects); objectsByGroup = createObjectsByGroup(allGroups, objectByName); @@ -46,7 +45,7 @@ public GroupInformation(Class clazz, List allObjects) { private Map createObjectByName(List allObjects) { Map objectsByName = new HashMap<>(); for (T object : allObjects) { - objectsByName.put(object.getName(), object); + objectsByName.put(object.toString(), object); } return objectsByName; } @@ -58,11 +57,11 @@ private Map> createObjectsByGroup(List allGroups, for (Group group : allGroups) { // Does this group contain any of the objects we're interested in today? - Sets.SetView objectsThisGroup = Sets.intersection(group.getMemberNames(), + Sets.SetView items = Sets.intersection(group.getItems(), objectByName.keySet()); - if (!objectsThisGroup.isEmpty()) { + if (!items.isEmpty()) { List objects = new ArrayList<>(); - for (String objectName : objectsThisGroup) { + for (String objectName : items) { objects.add(objectByName.get(objectName)); } groups.put(group, objects); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/GroupsDialog.java b/src/main/java/gov/nasa/ziggy/ui/util/GroupsDialog.java index 3e2c466..adf686b 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/GroupsDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/GroupsDialog.java @@ -1,9 +1,9 @@ package gov.nasa.ziggy.ui.util; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.NEW_SYMBOL; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.OK; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DELETE_SYMBOL; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.NEW_SYMBOL; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.OK; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.boldLabel; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; @@ -47,6 +47,8 @@ public class GroupsDialog extends javax.swing.JDialog { private static final Logger log = LoggerFactory.getLogger(GroupsDialog.class); + private String type; + private JCheckBox defaultCheckBox; private JTextField newGroupTextField; private JList groupsList; @@ -55,8 +57,9 @@ public class GroupsDialog extends javax.swing.JDialog { private final GroupOperations groupOperations = new GroupOperations(); - public GroupsDialog(Window owner) { + public GroupsDialog(Window owner, String type) { super(owner, DEFAULT_MODALITY_TYPE); + this.type = type; buildComponent(); setLocationRelativeTo(owner); @@ -81,7 +84,7 @@ private JPanel getDataPanel() { groupsListModel = new GenericListModel<>(); ArrayList groups = new ArrayList<>(); - groups.add(new Group(ZiggyGuiConstants.LOADING)); + groups.add(new Group(ZiggyGuiConstants.LOADING, ZiggyGuiConstants.LOADING)); groupsListModel.setList(groups); groupsList = new JList<>(groupsListModel); groupsList.addListSelectionListener(this::groupsListValueChanged); @@ -136,7 +139,7 @@ private void addGroup(ActionEvent evt) { } List currentList = groupsListModel.getList(); - Group newGroup = new Group(groupName); + Group newGroup = new Group(groupName, type); if (currentList.contains(newGroup)) { MessageUtils.showError(this, "A group by that name already exists"); return; @@ -211,7 +214,7 @@ private void loadFromDatabase(String selectedName) { new SwingWorker, Void>() { @Override protected List doInBackground() throws Exception { - return groupOperations().groups(); + return groupOperations().groups(type); } @Override @@ -242,8 +245,8 @@ private GroupOperations groupOperations() { return groupOperations; } - public static Group selectGroup(Component owner) { - GroupsDialog editor = new GroupsDialog(SwingUtilities.getWindowAncestor(owner)); + public static Group selectGroup(Component owner, String type) { + GroupsDialog editor = new GroupsDialog(SwingUtilities.getWindowAncestor(owner), type); editor.setVisible(true); if (editor.isCancelled) { @@ -256,6 +259,6 @@ public static Group selectGroup(Component owner) { } public static void main(String[] args) { - ZiggySwingUtils.displayTestDialog(new GroupsDialog((JFrame) null)); + ZiggySwingUtils.displayTestDialog(new GroupsDialog((JFrame) null, "Group Type")); } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/ParameterRendererFactory.java b/src/main/java/gov/nasa/ziggy/ui/util/ParameterRendererFactory.java index 9cc475d..e231412 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/ParameterRendererFactory.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/ParameterRendererFactory.java @@ -17,9 +17,9 @@ public class ParameterRendererFactory implements PropertyRendererFactory { /** - * Returns an appropriate {@link TableCellRenderer} for a {@link Parameter}. for arrays, - * the {@link ArrayTableCellRenderer} is returned. For scalar values, the default renderers from - * the {@link PropertyRendererRegistry} are used. + * Returns an appropriate {@link TableCellRenderer} for a {@link Parameter}. for arrays, the + * {@link ArrayTableCellRenderer} is returned. For scalar values, the default renderers from the + * {@link PropertyRendererRegistry} are used. */ @Override public TableCellRenderer createTableCellRenderer(Property property) { diff --git a/src/main/java/gov/nasa/ziggy/ui/util/PropertySheetHelper.java b/src/main/java/gov/nasa/ziggy/ui/util/PropertySheetHelper.java index 6ac1445..6f9eb72 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/PropertySheetHelper.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/PropertySheetHelper.java @@ -14,8 +14,8 @@ * Helper methods for working with L2fprod's {@link PropertySheetPanel}. *

* Note: if and when additional public static populatePropertySheet methods are added, the - * {@link Parameter#readFromObject(Object)} and {@link Parameter#writeToObject(Object)} - * must also be modified to support whatever class is used as the argument to the new method. + * {@link Parameter#readFromObject(Object)} and {@link Parameter#writeToObject(Object)} must also be + * modified to support whatever class is used as the argument to the new method. * * @author Todd Klaus * @author PT @@ -33,8 +33,7 @@ private PropertySheetHelper() { */ public static void populatePropertySheet(ParameterSet parameterSet, PropertySheetPanel propertySheetPanel) throws Exception { - propertySheetPanel - .setProperties(parameterSet.getParameters().toArray(new Parameter[0])); + propertySheetPanel.setProperties(parameterSet.getParameters().toArray(new Parameter[0])); propertySheetPanel.setRendererFactory(new ParameterRendererFactory()); propertySheetPanel.setEditorFactory(new ParameterEditorFactory()); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/TaskHalter.java b/src/main/java/gov/nasa/ziggy/ui/util/TaskHalter.java index 78a3d94..4565897 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/TaskHalter.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/TaskHalter.java @@ -2,11 +2,9 @@ import java.awt.Window; import java.util.Collection; -import java.util.HashMap; import java.util.List; -import java.util.Map; +import java.util.Set; import java.util.TreeSet; -import java.util.UUID; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; @@ -22,15 +20,15 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.services.messages.KilledTaskMessage; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; +import gov.nasa.ziggy.services.messages.TaskHaltedMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.instances.InstancesTasksPanel; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.Requestor; /** * Used to halt tasks. @@ -38,25 +36,23 @@ * @author PT * @author Bill Wohler */ -public class TaskHalter implements Requestor { +public class TaskHalter { private static final Logger log = LoggerFactory.getLogger(InstancesTasksPanel.class); private static final long AWAIT_DURATION_MILLIS = 15000L; - private final UUID uuid = UUID.randomUUID(); - private CountDownLatch replyMessagesCountdownLatch; - private Map tasksToHalt; + private Set tasksToHalt; - private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); public TaskHalter() { - // Subscribe to KilledTaskMessage so we can track whether all the tasks that + // Subscribe to TaskHaltedMessage so we can track whether all the tasks that // were marked for death, were actually halted. - ZiggyMessenger.subscribe(KilledTaskMessage.class, message -> { - if (isDestination(message)) { - tasksToHalt.remove(message.getTaskId()); + ZiggyMessenger.subscribe(TaskHaltedMessage.class, message -> { + if (tasksToHalt.contains(message.getPipelineTask())) { + tasksToHalt.remove(message.getPipelineTask()); if (replyMessagesCountdownLatch != null) { replyMessagesCountdownLatch.countDown(); } @@ -68,16 +64,16 @@ public TaskHalter() { * Halts running tasks with diagnostics going to stdout. * * @param pipelineInstance the pipeline instance containing the running tasks - * @param taskIdsToHalt the list of tasks to halt; may be null or empty to halt all running - * tasks in the instance + * @param tasksToHalt the list of tasks to halt; may be null or empty to halt all running tasks + * in the instance */ - public void haltTasks(PipelineInstance pipelineInstance, Collection taskIdsToHalt) { + public void haltTasks(PipelineInstance pipelineInstance, Collection tasksToHalt) { if (pipelineInstance == null) { return; } - haltTasks(null, runningTasksInInstance(pipelineInstance, taskIdsToHalt)); + haltTasks((Window) null, runningTasksInInstance(pipelineInstance, tasksToHalt)); } /** @@ -97,12 +93,12 @@ public void haltTasks(Window owner, PipelineInstance pipelineInstance) { * @param owner If non-null, the window for attaching dialogs; otherwise any warnings will * appear on stdout * @param pipelineInstance the pipeline instance containing the running tasks - * @param taskIdsToHalt the list of tasks to halt; may be null or empty to halt all running - * tasks in the instance + * @param tasksToHalt the list of tasks to halt; may be null or empty to halt all running tasks + * in the instance */ @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) public void haltTasks(Window owner, PipelineInstance pipelineInstance, - Collection taskIdsToHalt) { + Collection tasksToHalt) { // Are you sure you want to do this? if (pipelineInstance == null @@ -114,7 +110,7 @@ public void haltTasks(Window owner, PipelineInstance pipelineInstance, new SwingWorker, Void>() { @Override protected List doInBackground() throws Exception { - return runningTasksInInstance(pipelineInstance, taskIdsToHalt); + return runningTasksInInstance(pipelineInstance, tasksToHalt); } @Override @@ -129,20 +125,20 @@ protected void done() { } private List runningTasksInInstance(PipelineInstance pipelineInstance, - Collection taskIdsToHalt) { + Collection tasksToHalt) { // Collect all the processing pipeline tasks. - List incompleteTasksInSelectedInstance = pipelineTaskOperations() + List incompleteTasksInSelectedInstance = pipelineTaskDataOperations() .pipelineTasks(pipelineInstance, ProcessingStep.processingSteps()); // Check for selected tasks in list of running tasks in instance. - List taskIdsInSelectedInstance = incompleteTasksInSelectedInstance; - if (!CollectionUtils.isEmpty(taskIdsToHalt)) { - taskIdsInSelectedInstance = incompleteTasksInSelectedInstance.stream() - .filter(s -> taskIdsToHalt.contains(s.getId())) + List tasksInSelectedInstance = incompleteTasksInSelectedInstance; + if (!CollectionUtils.isEmpty(tasksToHalt)) { + tasksInSelectedInstance = incompleteTasksInSelectedInstance.stream() + .filter(t -> tasksToHalt.contains(t)) .collect(Collectors.toList()); } - return taskIdsInSelectedInstance; + return tasksInSelectedInstance; } private void haltTasks(Window owner, List runningTasksInInstance) { @@ -152,13 +148,13 @@ private void haltTasks(Window owner, List runningTasksInInstance) } // Attempt to halt the tasks. - tasksToHalt = new HashMap<>(); - for (PipelineTask task : runningTasksInInstance) { - tasksToHalt.put(task.getId(), task); - } + tasksToHalt = new TreeSet<>(runningTasksInInstance); replyMessagesCountdownLatch = new CountDownLatch(tasksToHalt.size()); - log.info("Halting tasks {}", new TreeSet<>(tasksToHalt.keySet())); - ZiggyMessenger.publish(KillTasksRequest.forTaskIds(this, tasksToHalt.keySet())); + log.info("Halting tasks {}", tasksToHalt); + for (PipelineTask pipelineTask : tasksToHalt) { + pipelineTaskDataOperations().setHaltRequested(pipelineTask, true); + } + ZiggyMessenger.publish(new HaltTasksRequest(tasksToHalt)); try { replyMessagesCountdownLatch.await(AWAIT_DURATION_MILLIS, TimeUnit.MILLISECONDS); log.info("Halting tasks...done"); @@ -171,13 +167,12 @@ private void haltTasks(Window owner, List runningTasksInInstance) // If any tasks were not halted within the timeout, update the log, generate // alerts, and let the user know. if (!tasksToHalt.isEmpty()) { - for (PipelineTask task : tasksToHalt.values()) { - String message = "Task " + task.getId() + " not halted after " + for (PipelineTask pipelineTask : tasksToHalt) { + String message = "Task " + pipelineTask + " not halted after " + AWAIT_DURATION_MILLIS / 1000L + " seconds"; log.warn(message); AlertService.getInstance() - .generateAndBroadcastAlert("PI", task.getId(), AlertService.Severity.WARNING, - message); + .generateAndBroadcastAlert("PI", pipelineTask, Severity.WARNING, message); } String message = "The " + (tasksToHalt.size() == 1 ? "task is" : "tasks are") + " still running, halting will continue in the background"; @@ -185,12 +180,7 @@ private void haltTasks(Window owner, List runningTasksInInstance) } } - @Override - public Object requestorIdentifier() { - return uuid; - } - - private PipelineTaskOperations pipelineTaskOperations() { - return pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/TaskRestarter.java b/src/main/java/gov/nasa/ziggy/ui/util/TaskRestarter.java index ab97610..0d939a0 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/TaskRestarter.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/TaskRestarter.java @@ -7,6 +7,7 @@ import java.util.Map; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; +import java.util.function.Function; import java.util.stream.Collectors; import javax.swing.SwingWorker; @@ -15,9 +16,12 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDisplayDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.messages.RestartTasksRequest; import gov.nasa.ziggy.services.messages.StartMemdroneRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; @@ -34,27 +38,29 @@ public class TaskRestarter { private final PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private final PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + private final PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); /** * Restarts stopped tasks with diagnostics going to stdout. * * @param pipelineInstance the pipeline instance containing the stopped tasks - * @param taskIdsToRestart the list of tasks to restart; may be null or empty to restart all + * @param tasksToRestart the list of tasks to restart; may be null or empty to restart all * stopped tasks in the instance * @param runMode the run mode to use when starting the tasks; if null, the user is prompted for * the mode using the non-null {@code owner} * @param messageSentLatch if non-null, the latch is decremented twice after the required * messages are published. */ - public void restartTasks(PipelineInstance pipelineInstance, Collection taskIdsToRestart, - RunMode runMode, CountDownLatch messageSentLatch) { + public void restartTasks(PipelineInstance pipelineInstance, + Collection tasksToRestart, RunMode runMode, CountDownLatch messageSentLatch) { if (pipelineInstance == null) { return; } restartTasks(null, runMode, messageSentLatch, - restartableTasksInInstance(pipelineInstance, taskIdsToRestart)); + restartableTasksInInstance(pipelineInstance, tasksToRestart)); } /** @@ -74,12 +80,12 @@ public void restartTasks(Window owner, PipelineInstance pipelineInstance) { * @param owner If non-null, the window for attaching dialogs; otherwise any warnings will * appear on stdout * @param pipelineInstance the pipeline instance containing the stopped tasks - * @param taskIdsToRestart the list of tasks to restart; may be null or empty to restart all + * @param tasksToRestart the list of tasks to restart; may be null or empty to restart all * stopped tasks in the instance */ public void restartTasks(Window owner, PipelineInstance pipelineInstance, - Collection taskIdsToRestart) { - restartTasks(owner, pipelineInstance, taskIdsToRestart, null, null); + Collection tasksToRestart) { + restartTasks(owner, pipelineInstance, tasksToRestart, null, null); } /** @@ -88,7 +94,7 @@ public void restartTasks(Window owner, PipelineInstance pipelineInstance, * @param owner If non-null, the window for attaching dialogs; otherwise any warnings will * appear on stdout * @param pipelineInstance the pipeline instance containing the stopped tasks - * @param taskIdsToRestart the list of tasks to restart; may be null or empty to restart all + * @param tasksToRestart the list of tasks to restart; may be null or empty to restart all * stopped tasks in the instance * @param runMode the run mode to use when starting the tasks; if null and {@code owner} is * non-null, the user is prompted for the mode; otherwise, @@ -98,16 +104,17 @@ public void restartTasks(Window owner, PipelineInstance pipelineInstance, */ @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) public void restartTasks(Window owner, PipelineInstance pipelineInstance, - Collection taskIdsToRestart, RunMode runMode, CountDownLatch messageSentLatch) { + Collection tasksToRestart, RunMode runMode, CountDownLatch messageSentLatch) { if (pipelineInstance == null) { return; } - new SwingWorker() { + new SwingWorker>, Void>() { @Override - protected RestartTasksInfo doInBackground() throws Exception { - return restartableTasksInInstance(pipelineInstance, taskIdsToRestart); + protected Map> doInBackground() + throws Exception { + return restartableTasksInInstance(pipelineInstance, tasksToRestart); } @Override @@ -121,24 +128,27 @@ protected void done() { }.execute(); } - private RestartTasksInfo restartableTasksInInstance(PipelineInstance pipelineInstance, - Collection taskIdsToRestart) { + private Map> restartableTasksInInstance( + PipelineInstance pipelineInstance, Collection tasksToRestart) { + // Collect all of the instance's tasks. List allTasksInSelectedInstance = pipelineTaskOperations() - .pipelineTasks(pipelineInstance, true); + .pipelineTasks(pipelineInstance); // Check for selected tasks in list of all tasks in instance. - List taskIdsInSelectedInstance = allTasksInSelectedInstance; - if (!CollectionUtils.isEmpty(taskIdsToRestart)) { - taskIdsInSelectedInstance = allTasksInSelectedInstance.stream() - .filter(s -> taskIdsToRestart.contains(s.getId())) + List tasksInSelectedInstance = allTasksInSelectedInstance; + if (!CollectionUtils.isEmpty(tasksToRestart)) { + tasksInSelectedInstance = allTasksInSelectedInstance.stream() + .filter(task -> tasksToRestart.contains(task)) .collect(Collectors.toList()); } // Filter the tasks to keep only the ones that are in the correct state for the // selected restart mode. - List failedTasks = new ArrayList<>(); - for (PipelineTask task : taskIdsInSelectedInstance) { + List pipelineTaskDisplayData = pipelineTaskDisplayDataOperations() + .pipelineTaskDisplayData(tasksInSelectedInstance); + List failedTasks = new ArrayList<>(); + for (PipelineTaskDisplayData task : pipelineTaskDisplayData) { if (checkTaskState(task)) { failedTasks.add(task); } @@ -148,9 +158,11 @@ private RestartTasksInfo restartableTasksInInstance(PipelineInstance pipelineIns } Map> supportedRunModesByPipelineTask = pipelineTaskOperations() - .supportedRunModesByPipelineTask(failedTasks); - - return new RestartTasksInfo(failedTasks, supportedRunModesByPipelineTask); + .supportedRunModesByPipelineTask( + failedTasks.stream().map(PipelineTaskDisplayData::getPipelineTask).toList()); + return failedTasks.stream() + .collect(Collectors.toMap(x -> x, + x -> supportedRunModesByPipelineTask.get(x.getPipelineTask()))); } /** @@ -158,56 +170,50 @@ private RestartTasksInfo restartableTasksInInstance(PipelineInstance pipelineIns * moment this is a pretty simple method, but once we add reruns of completed tasks it will get * more useful. */ - private boolean checkTaskState(PipelineTask task) { + private boolean checkTaskState(PipelineTaskDisplayData task) { return task.isError() || task.getProcessingStep() == ProcessingStep.COMPLETE && task.getFailedSubtaskCount() > 0; } private void restartTasks(Window owner, RunMode runMode, CountDownLatch messageSentLatch, - RestartTasksInfo restartTasksInfo) { - if (restartTasksInfo == null) { + Map> supportedRunModesByPipelineTaskDisplayData) { + + if (supportedRunModesByPipelineTaskDisplayData == null) { MessageUtils.showError(owner, "No restart-ready tasks found!", null); return; } RunMode restartMode = runMode != null ? runMode : owner != null - ? RestartDialog.restartTasks(owner, - restartTasksInfo.getSupportedRunModesByPipelineTask()) + ? RestartDialog.restartTasks(owner, supportedRunModesByPipelineTaskDisplayData) : RunMode.RESTART_FROM_BEGINNING; if (restartMode == null) { return; } - ZiggyMessenger.publish( - new RestartTasksRequest(restartTasksInfo.getFailedTasks(), false, restartMode), - messageSentLatch); - ZiggyMessenger.publish( - new StartMemdroneRequest(restartTasksInfo.getFailedTasks().get(0).getModuleName(), - restartTasksInfo.getFailedTasks().get(0).getPipelineInstanceId()), + List failedTasks = supportedRunModesByPipelineTaskDisplayData.keySet() + .stream() + .map( + (Function) PipelineTaskDisplayData::getPipelineTask) + .collect(Collectors.toList()); + for (PipelineTask pipelineTask : failedTasks) { + pipelineTaskDataOperations().setHaltRequested(pipelineTask, false); + } + ZiggyMessenger.publish(new RestartTasksRequest(failedTasks, false, restartMode), messageSentLatch); + ZiggyMessenger.publish(new StartMemdroneRequest(failedTasks.get(0).getModuleName(), + failedTasks.get(0).getPipelineInstanceId()), messageSentLatch); } private PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } - private static class RestartTasksInfo { - private List failedTasks; - private Map> supportedRunModesByPipelineTask; - - public RestartTasksInfo(List failedTasks, - Map> supportedRunModesByPipelineTask) { - this.failedTasks = failedTasks; - this.supportedRunModesByPipelineTask = supportedRunModesByPipelineTask; - } - - public List getFailedTasks() { - return failedTasks; - } + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations() { + return pipelineTaskDisplayDataOperations; + } - public Map> getSupportedRunModesByPipelineTask() { - return supportedRunModesByPipelineTask; - } + private PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java deleted file mode 100644 index fc1daa8..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java +++ /dev/null @@ -1,193 +0,0 @@ -package gov.nasa.ziggy.ui.util; - -import java.util.LinkedList; -import java.util.List; -import java.util.concurrent.ExecutionException; - -import javax.swing.JOptionPane; -import javax.swing.SwingUtilities; -import javax.swing.SwingWorker; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.services.config.KeyValuePair; -import gov.nasa.ziggy.services.config.KeyValuePairOperations; -import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; -import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; -import gov.nasa.ziggy.ui.util.models.DatabaseModel; -import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; - -/** - * Panel for displaying and editing arbitrary key/value pairs. - * - * @author Todd Klaus - * @author Bill Wohler - */ -@SuppressWarnings("serial") -public class ViewEditKeyValuePairPanel extends AbstractViewEditPanel { - private static Logger log = LoggerFactory.getLogger(ViewEditKeyValuePairPanel.class); - - private final KeyValuePairOperations keyValuePairOperations = new KeyValuePairOperations(); - - public ViewEditKeyValuePairPanel() { - super(new KeyValuePairTableModel()); - buildComponent(); - } - - @Override - protected void create() { - showEditDialog(new KeyValuePair()); - } - - @Override - protected void edit(int row) { - showEditDialog(ziggyTable.getContentAtViewRow(row)); - } - - private void showEditDialog(KeyValuePair keyValuePair) { - - EditKeyValuePairDialog dialog = new EditKeyValuePairDialog( - SwingUtilities.getWindowAncestor(this), keyValuePair); - dialog.setVisible(true); - - try { - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(this, e); - } - } - - @Override - protected void delete(int row) { - - KeyValuePair keyValuePair = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete key '" + keyValuePair.getKey() + "'?"); - - if (choice != JOptionPane.YES_OPTION) { - return; - } - new SwingWorker() { - @Override - protected Void doInBackground() throws Exception { - keyValuePairOperations().delete(keyValuePair); - return null; - } - - @Override - protected void done() { - try { - get(); - ziggyTable.loadFromDatabase(); - } catch (InterruptedException | ExecutionException e) { - MessageUtils.showError(ViewEditKeyValuePairPanel.this, e); - } - } - }.execute(); - } - - @Override - protected void refresh() { - try { - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtils.showError(this, e); - } - } - - private KeyValuePairOperations keyValuePairOperations() { - return keyValuePairOperations; - } - - private static class KeyValuePairTableModel extends AbstractZiggyTableModel - implements DatabaseModel { - - private static final String[] COLUMN_NAMES = { "Key", "Value" }; - - private List keyValuePairs = new LinkedList<>(); - - private final KeyValuePairOperations keyValuePairOperations = new KeyValuePairOperations(); - - public KeyValuePairTableModel() { - ZiggyMessenger.subscribe(InvalidateConsoleModelsMessage.class, this::invalidateModel); - } - - private void invalidateModel(InvalidateConsoleModelsMessage message) { - loadFromDatabase(); - } - - @Override - public void loadFromDatabase() { - new SwingWorker, Void>() { - @Override - protected List doInBackground() throws Exception { - return keyValuePairOperations().keyValuePairs(); - } - - @Override - protected void done() { - try { - keyValuePairs = get(); - fireTableDataChanged(); - } catch (InterruptedException | ExecutionException e) { - log.error("Can't retrieve key/value pairs", e); - } - } - }.execute(); - } - - // TODO Either find a use for getKeyValuePairAtRow or delete - @SuppressWarnings("unused") - public KeyValuePair getKeyValuePairAtRow(int rowIndex) { - return keyValuePairs.get(rowIndex); - } - - @Override - public int getRowCount() { - return keyValuePairs.size(); - } - - @Override - public int getColumnCount() { - return COLUMN_NAMES.length; - } - - @Override - public Object getValueAt(int rowIndex, int columnIndex) { - KeyValuePair keyValuePair = keyValuePairs.get(rowIndex); - - return switch (columnIndex) { - case 0 -> keyValuePair.getKey(); - case 1 -> keyValuePair.getValue(); - default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); - }; - } - - @Override - public Class getColumnClass(int columnIndex) { - return String.class; - } - - @Override - public String getColumnName(int column) { - return COLUMN_NAMES[column]; - } - - @Override - public KeyValuePair getContentAtRow(int row) { - return keyValuePairs.get(row); - } - - @Override - public Class tableModelContentClass() { - return KeyValuePair.class; - } - - private KeyValuePairOperations keyValuePairOperations() { - return keyValuePairOperations; - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/ZiggySwingUtils.java b/src/main/java/gov/nasa/ziggy/ui/util/ZiggySwingUtils.java index 2fb9b2e..6ade8bb 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/ZiggySwingUtils.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/ZiggySwingUtils.java @@ -17,6 +17,7 @@ import java.awt.event.MouseEvent; import java.awt.event.WindowAdapter; import java.awt.event.WindowEvent; +import java.lang.reflect.InvocationTargetException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -44,8 +45,10 @@ import com.jgoodies.looks.plastic.theme.SkyBluer; +import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.ui.util.table.TableMouseListener; import gov.nasa.ziggy.ui.util.table.ZiggyTable; +import gov.nasa.ziggy.util.os.OperatingSystemType; /** * A handful of Swing-related utilities. @@ -426,22 +429,53 @@ public static void ensureMinimumTableSize(ZiggyTable ziggyTable) { } /** - * Adjust the selection. Useful for popup triggers; for some reason the right mouse button - * doesn't modify the selection by default. + * Adjust the selection. This works better than the default table selection handler in two ways. + * It selects the row under the mouse if it isn't already when popping up a menu. It also avoids + * clearing the selection when showing the popup menu with Control-Click on the Mac. A table + * that supports multiple selection will want to run the following to remove the default table + * selection handler and call this method instead from {@code mousePressed()}. + * + * 

+     * for (MouseListener l : tasksTable.getTable().getMouseListeners()) {
+     *     if (l.getClass().getName().equals("javax.swing.plaf.basic.BasicTableUI$Handler")) {
+     *         tasksTable.getTable().removeMouseListener(l);
+     *     }
+     * }
+     *
*

* If the mouse is over a row that is already selected, then do not adjust the selection since * the leads to surprising behavior; otherwise, update the selection normally. * - * @param e the mouse event + * @param evt the mouse event */ - public static void adjustSelection(JTable table, MouseEvent e) { - int row = table.rowAtPoint(e.getPoint()); - if (table.isRowSelected(row)) { + public static void adjustSelection(JTable table, MouseEvent evt) { + int row = table.rowAtPoint(evt.getPoint()); + if (evt.isPopupTrigger() && table.isRowSelected(row)) { return; } - int column = table.columnAtPoint(e.getPoint()); - table.changeSelection(row, column, e.isControlDown(), e.isShiftDown()); + int column = table.columnAtPoint(evt.getPoint()); + table.changeSelection(row, column, + OperatingSystemType.newInstance() == OperatingSystemType.MAC_OS_X ? evt.isMetaDown() + : evt.isControlDown(), + evt.isShiftDown()); + } + + /** + * Waits for the EDT thread to process current events before proceeding. This is useful to + * separate methods such as {@code clearSelection()} that effectively call their listeners with + * {@code invokeLater()} and return immediately and code that changes the model. + */ + public static void flushEventDispatchThread() { + try { + SwingUtilities.invokeAndWait(() -> { + }); + } catch (InvocationTargetException e) { + throw new PipelineException("Error waiting for EDT to clear", e); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new PipelineException("Error waiting for EDT to clear", e); + } } /** diff --git a/src/main/java/gov/nasa/ziggy/ui/util/collections/ArrayImportExportUtils.java b/src/main/java/gov/nasa/ziggy/ui/util/collections/ArrayImportExportUtils.java index 9da1832..87d015e 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/collections/ArrayImportExportUtils.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/collections/ArrayImportExportUtils.java @@ -41,7 +41,7 @@ public static List importArray(File file) throws IOException { try { reader = new BufferedReader( new InputStreamReader(new FileInputStream(file), ZiggyFileUtils.ZIGGY_CHARSET)); - log.info("Importing array from: " + file.getName()); + log.info("Importing array from {}", file.getName()); String oneLine = null; @@ -60,7 +60,7 @@ public static List importArray(File file) throws IOException { } public static void exportArray(File file, List values) throws IOException { - log.info("Exporting array to: " + file.getName()); + log.info("Exporting array to {}", file.getName()); try (BufferedWriter writer = new BufferedWriter( new OutputStreamWriter(new FileOutputStream(file), ZiggyFileUtils.ZIGGY_CHARSET));) { diff --git a/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java b/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java index 6f40469..18c5f6a 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java @@ -19,7 +19,6 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.ui.util.GroupInformation; /** @@ -35,24 +34,22 @@ * @author PT * @author Bill Wohler */ -public class ZiggyTreeModel extends DefaultTreeModel { +public class ZiggyTreeModel extends DefaultTreeModel { private static final long serialVersionUID = 20240614L; private static final Logger log = LoggerFactory.getLogger(ZiggyTreeModel.class); - private static final String DEFAULT_GROUP_NAME = ""; - private final DefaultMutableTreeNode rootNode; private DefaultMutableTreeNode defaultGroupNode; private Map groupNodes; private Supplier> items; - private Class modelClass; + private String type; - public ZiggyTreeModel(Class modelClass, Supplier> items) { + public ZiggyTreeModel(String type, Supplier> items) { super(new DefaultMutableTreeNode("")); rootNode = (DefaultMutableTreeNode) getRoot(); - this.modelClass = modelClass; + this.type = type; this.items = items; } @@ -62,8 +59,8 @@ public void loadFromDatabase() throws PipelineException { @Override protected GroupInformation doInBackground() throws Exception { // Obtain information on the groups for this component class. - log.debug("Loading {} items", modelClass); - return new GroupInformation<>(modelClass, items.get()); + log.debug("Loading {} items", type); + return new GroupInformation<>(type, items.get()); } @Override @@ -71,12 +68,12 @@ protected void done() { GroupInformation groupInformation; try { groupInformation = get(); - log.debug("Loading {} items...done", modelClass); + log.debug("Loading {} items...done", type); // Add the default group. - log.debug("Updating tree model for {}", modelClass); + log.debug("Updating tree model for {}", type); rootNode.removeAllChildren(); - defaultGroupNode = new DefaultMutableTreeNode(DEFAULT_GROUP_NAME); + defaultGroupNode = new DefaultMutableTreeNode(Group.DEFAULT_NAME); insertNodeInto(defaultGroupNode, rootNode, rootNode.getChildCount()); Collections.sort(groupInformation.getObjectsInDefaultGroup(), @@ -111,9 +108,9 @@ protected void done() { reload(); - log.debug("Updating tree model for {}...done", modelClass); + log.debug("Updating tree model for {}...done", type); } catch (InterruptedException | ExecutionException e) { - log.error("Could not retrieve model data for {}", modelClass, e); + log.error("Could not retrieve model data for {}", type, e); } } }.execute(); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java index 80939b2..c9d0c43 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java @@ -17,11 +17,10 @@ import javax.swing.SwingUtilities; import javax.swing.SwingWorker; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.netbeans.swing.outline.RowModel; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.pipeline.definition.database.GroupOperations; import gov.nasa.ziggy.ui.util.GroupInformation; import gov.nasa.ziggy.ui.util.GroupsDialog; @@ -29,19 +28,14 @@ import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; /** - * Extension of {@link AbstractViewEditPanel} for classes of objects that extend {@link Groupable}, - * and hence need some mechanism by which their groups can be configured. - *

- * The addition of this class was necessary because I couldn't figure out a way to explain to the - * parent class that when we're using the grouping methods in {@link Groupable}; but otherwise, it - * might or might not. + * Extension of {@link AbstractViewEditPanel} that allows for the grouping of objects and for + * viewing in a tree control. * * @author PT */ -public abstract class AbstractViewEditGroupPanel - extends AbstractViewEditPanel { +public abstract class AbstractViewEditGroupPanel extends AbstractViewEditPanel { - private static final long serialVersionUID = 20240614L; + private static final long serialVersionUID = 20241004L; private final GroupOperations groupOperations = new GroupOperations(); @@ -88,11 +82,13 @@ public void actionPerformed(ActionEvent evt) { }); } + protected abstract String getType(); + /** * Assign objects in the table to a selected {@link Group}. */ - protected void group() { - Group group = GroupsDialog.selectGroup(this); + private void group() { + Group group = GroupsDialog.selectGroup(this, getType()); if (group == null) { return; } @@ -127,31 +123,28 @@ protected void done() { * Updates information in the new group and the groups that formerly held the objects in * question. */ - @SuppressWarnings("unchecked") private void updateGroups(List objects, Group group) { if (CollectionUtils.isEmpty(objects)) { return; } - Group databaseGroup = groupOperations().groupForName(group.getName(), - objects.get(0).getClass()); - GroupInformation groupInformation = new GroupInformation<>( - (Class) objects.get(0).getClass(), objects); + Group databaseGroup = groupOperations().group(group.getName(), group.getType()); + GroupInformation groupInformation = new GroupInformation<>(getType(), objects); for (T object : objects) { - Set memberNames = groupInformation.getGroupByObject() - .get(object) - .getMemberNames(); - if (!CollectionUtils.isEmpty(memberNames)) { - memberNames.remove(object.getName()); + Set items = groupInformation.getGroupByObject().get(object).getItems(); + if (!CollectionUtils.isEmpty(items)) { + items.remove(object.toString()); } if (databaseGroup != Group.DEFAULT) { - databaseGroup.getMemberNames().add(object.getName()); + databaseGroup.getItems().add(object.toString()); } } if (databaseGroup != Group.DEFAULT) { groupOperations().merge(databaseGroup); } - groupOperations().merge(groupInformation.getObjectsByGroup().keySet()); + Set groups = groupInformation.getObjectsByGroup().keySet(); + groups.remove(databaseGroup); // avoid merging stale information + groupOperations().merge(groups); } private GroupOperations groupOperations() { diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java index d0ea841..9b79ac1 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java @@ -1,22 +1,17 @@ package gov.nasa.ziggy.ui.util.table; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.COPY; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DELETE; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DIALOG; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EDIT; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.NEW; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.REFRESH; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.RENAME; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.VIEW; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createMenuItem; import java.awt.BorderLayout; -import java.awt.Component; import java.awt.Dimension; import java.awt.event.ActionEvent; -import java.awt.event.MouseAdapter; import java.awt.event.MouseEvent; +import java.awt.event.MouseListener; import java.util.ArrayList; import java.util.HashMap; import java.util.List; @@ -33,11 +28,12 @@ import javax.swing.ListSelectionModel; import javax.swing.SwingUtilities; import javax.swing.table.TableModel; +import javax.swing.tree.TreePath; import org.netbeans.swing.etable.ETable; +import org.netbeans.swing.outline.Outline; import org.netbeans.swing.outline.RowModel; -import gov.nasa.ziggy.pipeline.definition.UniqueNameVersionPipelineComponent; import gov.nasa.ziggy.ui.util.MessageUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; @@ -46,34 +42,23 @@ /** * Provides a framework for a Swing panel that includes a table and a button panel. The panel allows - * the user to edit the objects that are displayed in the table, create new objects, delete objects - * (unless they are locked), and to refresh the table contents. + * the user to edit the objects that are displayed in the table and to refresh the table contents. *

- * Subclasses of {@link AbstractViewEditPanel} provide a context menu that includes options to - * create a new object, edit an object, or delete an object. They also all include buttons to - * refresh the display and to create a new object. In addition, there are three optional actions - * that the panel can provide: copying an object, renaming an object, and assigning an object to a - * group. The selection of optional functions is controlled by the - * {@link #optionalViewEditFunctions()} method, which returns a {@link Set} of instances of - * {@link OptionalViewEditFunction}: - *

    - *
  1. A subclass will support copying table objects if the {@link #optionalViewEditFunctions()} - * returns a {@link Set} that includes {@link OptionalViewEditFunction#COPY}. In addition, the - * {@link #copy(int)} method must be overridden. A copy option will be added to the context menu. - *
  2. A subclass will support renaming table objects if the {@link #optionalViewEditFunctions()} - * returns a {@link Set} that includes {@link OptionalViewEditFunction#RENAME}. In addition, the - * {@link #rename(int)} method must be overridden. A rename option will be added to the context - * menu. - *
+ * Subclasses of {@link AbstractViewEditPanel} provide a context menu that includes options to view + * or edit an object. They also all include buttons to refresh the display. In addition, the panel + * can provide an optional action to assign an object to a group. The selection of optional + * functions is controlled by the {@link #optionalViewEditFunctions()} method, which returns a + * {@link Set} of instances of {@link OptionalViewEditFunction}: * * @author Todd Klaus * @author PT + * @author Bill Wohler */ @SuppressWarnings("serial") public abstract class AbstractViewEditPanel extends JPanel { public enum OptionalViewEditFunction { - REFRESH, NEW, VIEW, EDIT, COPY, RENAME, DELETE; + REFRESH, VIEW, EDIT; @Override public String toString() { @@ -102,8 +87,7 @@ public AbstractViewEditPanel(RowModel rowModel, ZiggyTreeModel treeModel, /** * Obtain the table from the {@code ZiggyTable} and set some general things. *

- * In particular, tables are set up with {@link ListSelectionModel#SINGLE_SELECTION} since - * almost none of the Ziggy tables are set up to work with multiple selections. The selection + * Tables are set up with {@link ListSelectionModel#SINGLE_SELECTION} by default. The selection * model can be updated if necessary. */ private ETable getTable(ZiggyTable ziggyTable) { @@ -134,18 +118,6 @@ public void actionPerformed(ActionEvent evt) { } }); - panelActions.put(OptionalViewEditFunction.NEW, new AbstractAction(NEW, null) { - @Override - public void actionPerformed(ActionEvent evt) { - try { - create(); - } catch (Exception e) { - MessageUtils - .showError(SwingUtilities.getWindowAncestor(AbstractViewEditPanel.this), e); - } - } - }); - panelActions.put(OptionalViewEditFunction.VIEW, new AbstractAction(VIEW, null) { @Override public void actionPerformed(ActionEvent evt) { @@ -170,52 +142,13 @@ public void actionPerformed(ActionEvent evt) { } }); - panelActions.put(OptionalViewEditFunction.COPY, new AbstractAction(COPY, null) { - @Override - public void actionPerformed(ActionEvent evt) { - try { - copy(selectedModelRow); - } catch (Exception e) { - MessageUtils - .showError(SwingUtilities.getWindowAncestor(AbstractViewEditPanel.this), e); - } - } - }); - - panelActions.put(OptionalViewEditFunction.RENAME, new AbstractAction(RENAME, null) { - @Override - public void actionPerformed(ActionEvent evt) { - try { - rename(selectedModelRow); - } catch (Exception e) { - MessageUtils - .showError(SwingUtilities.getWindowAncestor(AbstractViewEditPanel.this), e); - } - } - }); - - panelActions.put(OptionalViewEditFunction.DELETE, new AbstractAction(DELETE, null) { - @Override - public void actionPerformed(ActionEvent evt) { - try { - delete(selectedModelRow); - } catch (Exception e) { - MessageUtils - .showError(SwingUtilities.getWindowAncestor(AbstractViewEditPanel.this), e); - } - } - }); - return panelActions; } private JPanel getButtonPanel() { if (buttonPanel == null) { buttonPanel = ZiggySwingUtils.createButtonPanel(ButtonPanelContext.TOOL_BAR, null, - createButton(getActionByFunction().get(OptionalViewEditFunction.REFRESH)), - optionalViewEditFunctions().contains(OptionalViewEditFunction.NEW) - ? createButton(getActionByFunction().get(OptionalViewEditFunction.NEW)) - : null); + createButton(getActionByFunction().get(OptionalViewEditFunction.REFRESH))); } for (JButton button : buttons()) { ZiggySwingUtils.addButtonsToPanel(buttonPanel, button); @@ -238,86 +171,58 @@ protected List buttons() { } protected JScrollPane getScrollPane() { - JScrollPane scrollPane = new JScrollPane(table); - table.addMouseListener(new MouseAdapter() { + // Remove the existing table mouse listener as it clears the selection with Control-Click on + // the Mac rather than preserving the selection before it displays a menu. Perform selection + // ourselves using ZiggySwingUtils.adjustSelection(). + for (MouseListener l : table.getMouseListeners()) { + if (l.getClass().getName().equals("javax.swing.plaf.basic.BasicTableUI$Handler")) { + table.removeMouseListener(l); + } + } + table.addMouseListener(new java.awt.event.MouseAdapter() { + @Override + public void mousePressed(java.awt.event.MouseEvent evt) { + ZiggySwingUtils.adjustSelection(table, evt); + + updateActionState(actionByFunction); + + if (evt.isPopupTrigger()) { + displayPopupMenu(evt); + } else if (table instanceof Outline) { + toggleExpansionState((Outline) table, evt); + } + } + @Override public void mouseClicked(MouseEvent evt) { - tableMouseClicked(evt); + selectAndEditRow(evt); } }); - setComponentPopupMenu(table, getPopupMenu()); - return scrollPane; - } - private void tableMouseClicked(MouseEvent evt) { - int tableRow = table.rowAtPoint(evt.getPoint()); - selectedModelRow = table.convertRowIndexToModel(tableRow); - if (evt.getClickCount() == 2) { - try { - edit(selectedModelRow); - } catch (Exception e) { - MessageUtils.showError(SwingUtilities.getWindowAncestor(this), e); - } - } + return new JScrollPane(table); } /** - * Set the popup menu on the table. + * Updates the actions used in the buttons or context menu items. It is called any time the + * selection changes. */ - private void setComponentPopupMenu(final Component parent, final JPopupMenu menu) { - - parent.addMouseListener(new java.awt.event.MouseAdapter() { - @Override - public void mousePressed(java.awt.event.MouseEvent e) { - if (e.isPopupTrigger()) { - ZiggySwingUtils.adjustSelection(table, e); - selectedModelRow = table.convertRowIndexToModel(table.getSelectedRow()); - T contentAtViewRow = ziggyTable.getContentAtViewRow(selectedModelRow); - if (contentAtViewRow != null) { - if (contentAtViewRow instanceof UniqueNameVersionPipelineComponent) { - disableActionsWhenInUse(contentAtViewRow); - } - menu.show(parent, e.getX(), e.getY()); - } - } - } - - private void disableActionsWhenInUse(T contentAtViewRow) { - UniqueNameVersionPipelineComponent pipelineComponent = (UniqueNameVersionPipelineComponent) contentAtViewRow; - boolean inUse = pipelineComponent.getVersion() > 0 || pipelineComponent.isLocked(); - updateAction(OptionalViewEditFunction.RENAME, inUse); - updateAction(OptionalViewEditFunction.DELETE, inUse); - } + protected void updateActionState(Map actionByFunction) { + } - private void updateAction(OptionalViewEditFunction function, boolean locked) { - Action action = getActionByFunction().get(function); - action.setEnabled(!locked); - - // If we have a function that needs to end in "ed", then we'll do something - // different. - action.putValue(Action.SHORT_DESCRIPTION, - locked - ? "Components used in a pipeline run can not be " - + function.toString().toLowerCase() + "d" - : ""); - } - }); + private void displayPopupMenu(java.awt.event.MouseEvent evt) { + selectedModelRow = table.convertRowIndexToModel(table.getSelectedRow()); + T contentAtViewRow = ziggyTable.getContentAtViewRow(selectedModelRow); + if (contentAtViewRow != null) { + getPopupMenu().show(table, evt.getX(), evt.getY()); + } } private JPopupMenu getPopupMenu() { JPopupMenu popupMenu = new JPopupMenu(); - addOptionalMenuItem(popupMenu, OptionalViewEditFunction.NEW, - menuItem(OptionalViewEditFunction.NEW)); addOptionalMenuItem(popupMenu, OptionalViewEditFunction.VIEW, menuItem(OptionalViewEditFunction.VIEW)); popupMenu.add(menuItem(OptionalViewEditFunction.EDIT)); - addOptionalMenuItem(popupMenu, OptionalViewEditFunction.COPY, - menuItem(OptionalViewEditFunction.COPY)); - addOptionalMenuItem(popupMenu, OptionalViewEditFunction.RENAME, - menuItem(OptionalViewEditFunction.RENAME)); - addOptionalMenuItem(popupMenu, OptionalViewEditFunction.DELETE, - menuItem(OptionalViewEditFunction.DELETE)); for (JMenuItem menuItem : menuItems()) { popupMenu.add(menuItem); @@ -332,8 +237,12 @@ private void addOptionalMenuItem(JPopupMenu popupMenu, OptionalViewEditFunction } } + /** + * Returns a list of the {@link OptionalViewEditFunction}s that the panel supports. By default, + * none are provided. + */ protected Set optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.NEW); + return Set.of(); } private JMenuItem menuItem(OptionalViewEditFunction function) { @@ -351,23 +260,40 @@ protected List menuItems() { return new ArrayList<>(); } - protected abstract void refresh(); - - protected abstract void create(); - - protected void view(int row) { + private void toggleExpansionState(Outline outline, java.awt.event.MouseEvent evt) { + TreePath treePath = outline.getClosestPathForLocation(evt.getPoint().x, evt.getPoint().y); + if (outline.isExpanded(treePath)) { + outline.collapsePath(treePath); + } else { + outline.expandPath(treePath); + } } - protected abstract void edit(int row); - - protected void copy(int row) { + private void selectAndEditRow(MouseEvent evt) { + int tableRow = table.rowAtPoint(evt.getPoint()); + selectedModelRow = table.convertRowIndexToModel(tableRow); + if (evt.getClickCount() == 2) { + try { + if (ziggyTable.getContentAtViewRow(selectedModelRow) != null) { + edit(selectedModelRow); + } + } catch (Exception e) { + MessageUtils.showError(SwingUtilities.getWindowAncestor(this), e); + } + } } - protected void rename(int row) { + protected abstract void refresh(); + + protected void view(int row) { } - protected abstract void delete(int row); + protected abstract void edit(int row); + /** + * Returns a map of {@link Action} by {@link OptionalViewEditFunction}, which can be used to + * enable or disable buttons or menu items, for example. + */ private Map getActionByFunction() { return actionByFunction; } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/TableUpdater.java b/src/main/java/gov/nasa/ziggy/ui/util/table/TableUpdater.java new file mode 100644 index 0000000..01c540a --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/TableUpdater.java @@ -0,0 +1,100 @@ +package gov.nasa.ziggy.ui.util.table; + +import java.util.List; + +import javax.swing.table.AbstractTableModel; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Updates or inserts table rows if the data has changed. To use, return this object from + * {@code SwingWorker.doInBackground()} and call the {@link #updateTable(AbstractTableModel)} method + * in {@code SwingWorker.done()}. + */ +public class TableUpdater { + + private static final Logger log = LoggerFactory.getLogger(TableUpdater.class); + + private List insertedRows; + private List updatedRows; + private List deletedRows; + + /** + * Creates an {@code InstanceEventInfoComparer} object. + * + * @param previousElements a list of objects from the previous update that must have a + * reasonable {@link #equals(Object)} method; if null, all currentElements are considered new + * @param currentElements a list of objects for this update that must have a reasonable + * {@link #equals(Object)} method; if null, all existing rows will be deleted + */ + public TableUpdater(List previousElements, List currentElements) { + log.trace("previousElements={}", previousElements); + log.trace("currentElements={}", currentElements); + + int previousCount = previousElements == null ? 0 : previousElements.size(); + int currentCount = currentElements == null ? 0 : currentElements.size(); + + // Check for a change in row count. + if (currentCount == previousCount) { + insertedRows = List.of(); + deletedRows = List.of(); + } else if (currentCount > previousCount) { + insertedRows = List.of(previousCount, currentCount - 1); + deletedRows = List.of(); + } else { + insertedRows = List.of(); + deletedRows = List.of(currentCount, previousCount - 1); + } + + // Check for updated rows. + int minUpdatedRow = -1; + int maxUpdatedRow = -1; + for (int i = 0; i < Math.min(previousCount, currentCount); i++) { + boolean elementsEqual = previousElements.get(i).equals(currentElements.get(i)); + if (minUpdatedRow == -1 && !elementsEqual) { + minUpdatedRow = i; + } + if (!elementsEqual) { + maxUpdatedRow = i; + } + } + updatedRows = minUpdatedRow == -1 ? List.of() : List.of(minUpdatedRow, maxUpdatedRow); + } + + public void updateTable(AbstractTableModel tableModel) { + if (getDeletedRows().size() > 0) { + log.debug("Deleting rows {} in {}", getDeletedRows(), + tableModel.getClass().getSimpleName()); + for (int i = getDeletedRows().get(0); i <= getDeletedRows().get(1); i++) { + tableModel.fireTableRowsDeleted(i, i); + } + } + if (getInsertedRows().size() > 0) { + log.debug("Inserting rows {} in {}", getInsertedRows(), + tableModel.getClass().getSimpleName()); + for (int i = getInsertedRows().get(0); i <= getInsertedRows().get(1); i++) { + tableModel.fireTableRowsInserted(i, i); + } + } + if (getUpdatedRows().size() > 0) { + log.debug("Updating rows {} in {}", getUpdatedRows(), + tableModel.getClass().getSimpleName()); + for (int i = getUpdatedRows().get(0); i <= getUpdatedRows().get(1); i++) { + tableModel.fireTableRowsUpdated(i, i); + } + } + } + + private List getDeletedRows() { + return deletedRows; + } + + private List getInsertedRows() { + return insertedRows; + } + + private List getUpdatedRows() { + return updatedRows; + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java b/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java index 8a15119..aa20492 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java @@ -29,13 +29,11 @@ import javax.swing.JTextArea; import javax.swing.event.TreeModelEvent; import javax.swing.event.TreeModelListener; -import javax.swing.table.AbstractTableModel; import javax.swing.table.DefaultTableCellRenderer; import javax.swing.table.TableCellRenderer; import javax.swing.table.TableColumn; import javax.swing.table.TableColumnModel; import javax.swing.table.TableModel; -import javax.swing.tree.AbstractLayoutCache; import javax.swing.tree.DefaultMutableTreeNode; import javax.swing.tree.TreeModel; import javax.swing.tree.TreeNode; @@ -51,7 +49,6 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.models.DatabaseModel; @@ -116,7 +113,7 @@ public class ZiggyTable { private WrappingCellRenderer wrappingCellRenderer = new WrappingCellRenderer(); private DateCellRenderer dateCellRenderer = new DateCellRenderer(); private Map expansionState; - private boolean defaultGroupExpansionState; + private boolean defaultGroupExpansionState = true; private TableModel tableModel; private ZiggyTreeModel treeModel; private final Class modelContentsClass; @@ -147,8 +144,6 @@ public ZiggyTable(RowModel rowModel, ZiggyTreeModel treeModel, String nodesCo checkArgument(rowModel instanceof ModelContentClass, "ZiggyTable rowModel must implement ModelContentClass"); modelContentsClass = ((ModelContentClass) rowModel).tableModelContentClass(); - checkArgument(Groupable.class.isAssignableFrom(modelContentsClass), - "ZiggyTable model content class must extend Groupable"); this.treeModel = treeModel; table = new ZiggyOutline(); @@ -164,19 +159,22 @@ public ZiggyTable(RowModel rowModel, ZiggyTreeModel treeModel, String nodesCo @Override public void treeStructureChanged(TreeModelEvent evt) { - log.debug("thread={}, event={}", Thread.currentThread().getName(), evt); - if (expansionState == null) { - // This is the first load of the model. - return; + log.debug( + "thread={}, event={}, rootNode={}, defaultGroupNode={}, expansionState={}", + Thread.currentThread().getName(), evt, treeModel.getRootNode(), + treeModel.getDefaultGroupNode(), expansionState); + + if (treeModel.getDefaultGroupNode() != null) { + Outline outline = (Outline) table; + TreePath treePath = new TreePath(treeModel.getDefaultGroupNode().getPath()); + if (defaultGroupExpansionState) { + outline.expandPath(treePath); + } else { + outline.collapsePath(treePath); + } } - applyExpansionState(treeModel.getGroupNodes(), expansionState); - Outline outline = (Outline) table; - if (defaultGroupExpansionState) { - outline.expandPath(new TreePath(treeModel.getDefaultGroupNode().getPath())); - } else { - outline.collapsePath(new TreePath(treeModel.getDefaultGroupNode().getPath())); - } + applyExpansionState(treeModel.getGroupNodes(), expansionState); } @Override @@ -232,9 +230,9 @@ public void loadFromDatabase() { if (table instanceof Outline) { expansionState = currentExpansionState(treeModel); if (treeModel.getDefaultGroupNode() != null) { - // Null until the model has been loaded yet. - defaultGroupExpansionState = ((Outline) table).getLayoutCache() - .isExpanded(new TreePath(treeModel.getDefaultGroupNode())); + // Null until the model has been loaded. + defaultGroupExpansionState = ((Outline) table) + .isExpanded(new TreePath(treeModel.getDefaultGroupNode().getPath())); } treeModel.loadFromDatabase(); @@ -263,11 +261,10 @@ private Map currentExpansionState(ZiggyTreeModel treeModel) Map expansionState = new HashMap<>(); Outline outline = (Outline) table; - AbstractLayoutCache layoutCache = outline.getLayoutCache(); for (String groupName : groupNodes.keySet()) { DefaultMutableTreeNode node = groupNodes.get(groupName); - boolean isExpanded = layoutCache.isExpanded(new TreePath(node.getPath())); + boolean isExpanded = outline.isExpanded(new TreePath(node.getPath())); expansionState.put(groupName, isExpanded); } @@ -283,6 +280,10 @@ private void applyExpansionState(Map groupNodes, Map expansionState) { checkState(table instanceof Outline, "table must be an Outline"); + if (expansionState == null) { + return; + } + Outline outline = (Outline) table; for (String groupName : expansionState.keySet()) { @@ -399,12 +400,6 @@ public T getContentAtViewRow(int viewIndex) { : getTableContentAtModelRow(modelIndex); } - public void fireTableDataChanged() { - if (tableModel instanceof AbstractTableModel) { - ((AbstractTableModel) tableModel).fireTableDataChanged(); - } - } - /** * Returns row content from an instance of {@link Outline}. */ @@ -612,7 +607,7 @@ private TableCellRenderer getCellRenderer(int row, int column, * * @author PT */ - class ZiggyETable extends ETable { + private class ZiggyETable extends ETable { private static final long serialVersionUID = 20230511L; @@ -635,7 +630,7 @@ public TableCellRenderer getCellRenderer(int row, int column) { * * @author PT */ - class ZiggyOutline extends Outline { + private class ZiggyOutline extends Outline { private static final long serialVersionUID = 20230511L; diff --git a/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java index fec8b82..2917dcd 100644 --- a/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java @@ -58,8 +58,8 @@ public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanc // will be the only unit of work. if (directoryContainsManifest(rootDirectory())) { UnitOfWork uow = new UnitOfWork(); - uow.addParameter(new Parameter(DIRECTORY_PARAMETER_NAME, - rootDirectory().toString(), ZiggyDataType.ZIGGY_STRING)); + uow.addParameter(new Parameter(DIRECTORY_PARAMETER_NAME, rootDirectory().toString(), + ZiggyDataType.ZIGGY_STRING)); unitsOfWork.add(uow); } else { diff --git a/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java index ebfe355..7cd8a26 100644 --- a/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java @@ -10,7 +10,7 @@ import java.util.Map; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.data.datastore.DataFileType; @@ -79,7 +79,7 @@ public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanc // Generate the UOWs from the BriefStatePathInformation instances. List unitsOfWork = new ArrayList<>(); for (UowPathInformation unitOfWorkPathInformation : unitsOfWorkPathInformation) { - UnitOfWork uow = new UnitOfWork(); + UnitOfWork uow = new UnitOfWork(unitOfWorkPathInformation.getBriefState()); // Populate the UOW parameters for the datastore paths used by this UOW. populateDirectoryParameters(uow, unitOfWorkPathInformation); @@ -91,7 +91,6 @@ public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanc uow.addParameter(new Parameter(regexpEntry.getKey(), regexpEntry.getValue(), ZiggyDataType.ZIGGY_STRING)); } - uow.setBriefState(unitOfWorkPathInformation.getBriefState()); unitsOfWork.add(uow); } return unitsOfWork; diff --git a/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java b/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java index ffcdae7..1486769 100644 --- a/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java +++ b/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java @@ -1,20 +1,28 @@ package gov.nasa.ziggy.uow; +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + import java.io.Serializable; import java.util.Collection; -import java.util.HashMap; import java.util.HashSet; -import java.util.Map; import java.util.Set; +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.lang3.StringUtils; + import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.pipeline.definition.Parameter; +import jakarta.persistence.ElementCollection; +import jakarta.persistence.Embeddable; +import jakarta.persistence.FetchType; +import jakarta.persistence.JoinTable; /** * Defines the unit of work to be processed by a specified task. *

- * The unit of work contains a set of parameters, in the form of {@link Parameter} instances, - * that can be used by a pipeline module to determine the range of data to be processed by a given + * The unit of work contains a set of parameters, in the form of {@link Parameter} instances, that + * can be used by a pipeline module to determine the range of data to be processed by a given * pipeline task. For example, for units of work based on time range, the parameters might be the * start time and stop time for the task. *

@@ -29,40 +37,65 @@ * brief state value to be set for the unit of work. * * @author PT + * @author Bill Wohler */ +@Embeddable public class UnitOfWork implements Serializable, Comparable { - private static final long serialVersionUID = 20230511L; + private static final long serialVersionUID = 20240819L; public static final String BRIEF_STATE_PARAMETER_NAME = "briefState"; - private Map parameterByName = new HashMap<>(); + @ElementCollection(fetch = FetchType.EAGER) + @JoinTable(name = "ziggy_UnitOfWork_parameters") + private Set parameters = new HashSet<>(); + + public UnitOfWork() { + } + + public UnitOfWork(String briefState) { + setBriefState(briefState); + } public String briefState() { return getParameter(BRIEF_STATE_PARAMETER_NAME).getString(); } public void setBriefState(String briefState) { + checkNotNull(briefState, "briefState"); + checkArgument(!StringUtils.isBlank(briefState), "briefState can't be empty"); addParameter( new Parameter(BRIEF_STATE_PARAMETER_NAME, briefState, ZiggyDataType.ZIGGY_STRING)); } public void addParameter(Parameter parameter) { - parameterByName.put(parameter.getName(), parameter); + checkNotNull(parameter, "parameter"); + if (!parameters.add(parameter)) { + // A parameter with the same name already exists. + parameters.remove(parameter); + parameters.add(parameter); + } } public Parameter getParameter(String name) { - return parameterByName.get(name); + for (Parameter parameter : parameters) { + if (parameter.getName().equals(name)) { + return parameter; + } + } + return null; } public Set getParameters() { - return new HashSet<>(parameterByName.values()); + return new HashSet<>(parameters); } public void setParameters(Collection parameters) { - parameterByName.clear(); + checkNotNull(parameters, "parameters"); + checkArgument(!CollectionUtils.isEmpty(parameters), "parameters can't be empty"); + this.parameters.clear(); for (Parameter parameter : parameters) { - parameterByName.put(parameter.getName(), parameter); + addParameter(parameter); } } diff --git a/src/main/java/gov/nasa/ziggy/util/BuildInfo.java b/src/main/java/gov/nasa/ziggy/util/BuildInfo.java new file mode 100644 index 0000000..395e31d --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/util/BuildInfo.java @@ -0,0 +1,256 @@ +package gov.nasa.ziggy.util; + +import static com.google.common.base.Preconditions.checkNotNull; + +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.UncheckedIOException; +import java.io.Writer; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; + +import org.apache.commons.cli.DefaultParser; +import org.apache.commons.cli.Option; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.ParseException; +import org.apache.commons.configuration2.PropertiesConfiguration; +import org.apache.commons.configuration2.builder.fluent.Configurations; +import org.apache.commons.configuration2.ex.ConfigurationException; +import org.apache.commons.exec.CommandLine; + +import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.config.ZiggyConfiguration; +import gov.nasa.ziggy.services.process.ExternalProcess; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.ZiggyFileUtils; + +/** + * Generates and manages Git information about the version of Ziggy itself, or of a pipeline that + * runs under Ziggy. The information is written to a properties file for use by the + * {@link ZiggyConfiguration} API. For Ziggy, the file is named ziggy-build.properties and is saved + * to ziggy's build/etc directory. For a pipeline, the file is named pipeline-build.properties and + * is saved to the etc directory under the pipeline's home directory, as defined by the + * ziggy.pipeline.home.dir property. + *

+ * When called from the command line, the default behavior of {@link BuildInfo} is to produce the + * pipeline version information file. The user can optionally specify "ZIGGY" or "PIPELINE" at the + * command line to specify which version information to produce. The command line argument is not + * case-sensitive. + * + * @author PT + */ +public class BuildInfo { + + public enum BuildType { + ZIGGY(PropertyName.ZIGGY_HOME_DIR, "ziggy.", "ziggy-build.properties"), + PIPELINE(PropertyName.PIPELINE_HOME_DIR, "pipeline.", "pipeline-build.properties"); + + private PropertyName homeDirPropertyName; + private String propertyPrefix; + private String buildInfoFilename; + + BuildType(PropertyName homeDirPropertyName, String propertyPrefix, + String buildInfoFilename) { + this.homeDirPropertyName = homeDirPropertyName; + this.propertyPrefix = propertyPrefix; + this.buildInfoFilename = buildInfoFilename; + } + + public PropertyName homeDirPropertyName() { + return homeDirPropertyName; + } + + public String propertyPrefix() { + return propertyPrefix; + } + + public String buildInfoFilename() { + return buildInfoFilename; + } + } + + private static final int ABBREV = 10; + private static final String GIT_COMMAND = "git"; + static final List VERSION_ARGS = List.of("describe", "--always", "--abbrev=" + ABBREV); + static final List BRANCH_ARGS = List.of("rev-parse", "--abbrev-ref", "HEAD"); + static final List COMMIT_ARGS = List.of("rev-parse", "--short=" + ABBREV, "HEAD"); + private static final String HEADER = "# This file is automatically generated by Ziggy." + + System.lineSeparator() + "# Do not edit." + System.lineSeparator(); + private static final String VERSION_DIRECTORY_NAME = "etc"; + private static final String BUILD_VERSION_PROPERTY_NAME_SUFFIX = "version"; + private static final String BUILD_BRANCH_PROPERTY_NAME_SUFFIX = "version.branch"; + private static final String BUILD_COMMIT_PROPERTY_NAME_SUFFIX = "version.commit"; + private static final String PREFIX_OPTION = "prefix"; + private static final String HOME_DIR_OPTION = "home"; + public static final String MISSING_PROPERTY_VALUE = "None provided"; + + private final BuildType buildType; + private String homeDir; + + /** + * Constructs a {@code BuildInfo} object of the given type that obtains the location of the home + * directory from the type's homeDirPropertyName property. + */ + public BuildInfo(BuildType buildType) { + this(buildType, ZiggyConfiguration.getInstance() + .getString(buildType.homeDirPropertyName().property(), null)); + } + + /** + * Constructs a {@code BuildInfo} object of the given type if properties are not available, as + * is the case when compiling. + * + * @param buildType the non-null {@link BuildType} for this object + * @param homeDir the home directory where + * {@value #VERSION_DIRECTORY_NAME}/buildType.homeDirPropertyName will be written or + * read; null is allowed for limited use, but the behavior is undefined + */ + private BuildInfo(BuildType buildType, String homeDir) { + this.buildType = checkNotNull(buildType, "buildType"); + this.homeDir = homeDir; + } + + private Path versionDir() { + return homeDir != null ? Paths.get(homeDir, VERSION_DIRECTORY_NAME) : null; + } + + private Path versionFile() { + Path versionDir = versionDir(); + return versionDir != null ? versionDir.resolve(buildType.buildInfoFilename()) : null; + } + + private String buildVersionPropertyName() { + return buildType.propertyPrefix() + BUILD_VERSION_PROPERTY_NAME_SUFFIX; + } + + private String buildBranchPropertyName() { + return buildType.propertyPrefix() + BUILD_BRANCH_PROPERTY_NAME_SUFFIX; + } + + private String buildCommitPropertyName() { + return buildType.propertyPrefix() + BUILD_COMMIT_PROPERTY_NAME_SUFFIX; + } + + /** Returns the full output of {@code git describe}. */ + public String getSoftwareVersion() { + return propertyValue(buildVersionPropertyName()); + } + + private String propertyValue(String property) { + return buildType == BuildType.ZIGGY ? configuration().getString(property) + : configuration().getString(property, MISSING_PROPERTY_VALUE); + } + + /** Returns true if the output of {@code git describe} only contains the tag. */ + public boolean isRelease() { + return !getSoftwareVersion().endsWith(getRevision()); + } + + /** Returns the current branch. */ + public String getBranch() { + return propertyValue(buildBranchPropertyName()); + } + + /** Returns the current commit. */ + public String getRevision() { + return propertyValue(buildCommitPropertyName()); + } + + /** Loads the properties file to an instance of {@link PropertiesConfiguration}. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private PropertiesConfiguration configuration() { + try { + Path versionFile = versionFile(); + PropertiesConfiguration configuration = versionFile != null && Files.exists(versionFile) + ? new Configurations().properties(versionFile.toFile()) + : new PropertiesConfiguration(); + configuration.setThrowExceptionOnMissing(true); + return configuration; + } catch (ConfigurationException e) { + throw new PipelineException(e); + } + } + + public void writeBuildFile() { + + try { + Files.createDirectories(versionDir()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + + File versionFile = versionFile().toFile(); + try ( + Writer fstream = new OutputStreamWriter(new FileOutputStream(versionFile), + ZiggyFileUtils.ZIGGY_CHARSET); + BufferedWriter output = new BufferedWriter(fstream)) { + output.write(HEADER); + output.write(buildVersionPropertyName() + " = " + + externalProcessResults(VERSION_ARGS).get(0) + System.lineSeparator()); + output.write(buildBranchPropertyName() + " = " + + externalProcessResults(BRANCH_ARGS).get(0) + System.lineSeparator()); + output.write(buildCommitPropertyName() + " = " + + externalProcessResults(COMMIT_ARGS).get(0) + System.lineSeparator()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + List externalProcessResults(List gitArgs) { + CommandLine commandLine = new CommandLine(GIT_COMMAND); + for (String gitArg : gitArgs) { + commandLine.addArgument(gitArg); + } + ExternalProcess externalProcess = ExternalProcess.simpleExternalProcess(commandLine); + externalProcess.logStdErr(false); + externalProcess.logStdOut(false); + externalProcess.writeStdErr(true); + externalProcess.writeStdOut(true); + externalProcess.setWorkingDirectory(versionDir().toFile()); + externalProcess.run(true, 0); + return externalProcess.stdout(); + } + + /** Convenience method for getting the Ziggy software version. */ + public static String ziggyVersion() { + return new BuildInfo(BuildType.ZIGGY).getSoftwareVersion(); + } + + /** Convenience method for getting the pipeline software version. */ + public static String pipelineVersion() { + return new BuildInfo(BuildType.PIPELINE).getSoftwareVersion(); + } + + /** Generates the properties file for the pipeline. */ + public static void main(String[] args) throws ParseException { + + String prefix = "pipeline"; + Options options = new Options() + .addOption(Option.builder("p") + .longOpt(PREFIX_OPTION) + .hasArg() + .desc("Set property and filename prefix (default: " + prefix + ")") + .build()) + .addOption(Option.builder("h") + .longOpt(HOME_DIR_OPTION) + .hasArg() + .desc("Destination home directory for " + VERSION_DIRECTORY_NAME + + "/-build.properties") + .required() + .build()); + + org.apache.commons.cli.CommandLine commandLine = new DefaultParser().parse(options, args); + if (commandLine.hasOption(PREFIX_OPTION)) { + prefix = commandLine.getOptionValue(PREFIX_OPTION); + } + BuildType versionType = BuildType.valueOf(prefix.toUpperCase()); + + new BuildInfo(versionType, commandLine.getOptionValue(HOME_DIR_OPTION)).writeBuildFile(); + } +} diff --git a/src/main/java/gov/nasa/ziggy/util/ClasspathScanner.java b/src/main/java/gov/nasa/ziggy/util/ClasspathScanner.java index bd47987..375983b 100644 --- a/src/main/java/gov/nasa/ziggy/util/ClasspathScanner.java +++ b/src/main/java/gov/nasa/ziggy/util/ClasspathScanner.java @@ -72,7 +72,7 @@ private void notifyListeners(ClassFile classFile) { * a JAR scan) for annotated classes. */ public void scanForClasses() { - log.debug("ClasspathScanner: Scanning class path for matching classes"); + log.debug("Scanning class path for matching classes"); visitedClassPathElements = new HashSet<>(); Set classPathElements = classPathToScan; @@ -96,7 +96,7 @@ private void scanClassPath(Set classPath) { File classPathElementFile = new File(classPathElement); if (classPathElementFile.exists()) { if (classPathElementFile.isDirectory()) { - log.debug("Scanning directory {}" + classPathElementFile); + log.debug("Scanning directory {}", classPathElementFile); scanDirectory(classPathElementFile, classPathElementFile); } else if (classPathElementFile.getName().endsWith(".jar")) { if (matchesJarFilter(classPathElementFile.getName())) { @@ -129,7 +129,7 @@ private void scanDirectory(File rootDirectory, File directory) { String fullyQualifiedName = convertPathToPackageName(relativePath); if (matchesPackageFilter(fullyQualifiedName) && fullyQualifiedName.endsWith(".class")) { - log.debug("Processing file={}", file.getName()); + log.debug("Processing {}", file.getName()); FileInputStream fis; try { fis = new FileInputStream(file); @@ -161,7 +161,7 @@ private void scanJar(File jarFile) { if (mainAttrs != null) { String manifestClassPath = mainAttrs.getValue(Attributes.Name.CLASS_PATH); if (manifestClassPath != null) { - log.debug("Found MANIFEST ClassPath={}", manifestClassPath); + log.debug("Found MANIFEST, ClassPath={}", manifestClassPath); String classPathRelativeDir = jarFile.getParentFile().getAbsolutePath(); String[] classPathEntries = manifestClassPath.split("\\s+"); @@ -184,7 +184,7 @@ private void scanJar(File jarFile) { String fullyQualifiedName = convertPathToPackageName(entry.getName()); if (matchesPackageFilter(fullyQualifiedName) && fullyQualifiedName.endsWith(".class")) { - log.debug("Processing entry={}", entry); + log.debug("Processing entry {}", entry); processFile(jar.getInputStream(entry)); } } @@ -224,11 +224,11 @@ private Set parseClassPath() { ClassLoader classLoader = getClass().getClassLoader(); if (classLoader instanceof URLClassLoader) { - log.debug("classLoader={} instanceof URLClassLoader", classLoader); + log.debug("Class loader {} instanceof URLClassLoader", classLoader); URL[] urls = ((URLClassLoader) classLoader).getURLs(); for (URL url : urls) { String filename = url.getFile(); - log.debug("Adding url={}", filename); + log.debug("Adding URL {}", filename); classPath.add(filename); } } else { @@ -238,7 +238,7 @@ private Set parseClassPath() { File.pathSeparator); while (st.hasMoreTokens()) { String filename = st.nextToken(); - log.debug("Adding file={}", filename); + log.debug("Adding {}", filename); classPath.add(filename); } } diff --git a/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java b/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java deleted file mode 100644 index 26bf2ae..0000000 --- a/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java +++ /dev/null @@ -1,120 +0,0 @@ -package gov.nasa.ziggy.util; - -import java.util.ArrayList; -import java.util.Date; -import java.util.List; - -import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; - -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.util.dispmod.DisplayModel; - -/** - * Computes the following statistics based on the processing times for the specified list of - * pipeline tasks: - * - * 

- * max
- * min
- * mean
- * stddev
- *
- * - * @author Todd Klaus - */ -public class TaskProcessingTimeStats { - private int count; - private double sum; - private double min; - private double max; - private double mean; - private double stddev; - private Date minStart = new Date(); - private Date maxEnd = new Date(0); - private double totalElapsed; - - /** - * Private to prevent instantiation. Use static 'of' method to create instances. - * - * @param tasks - */ - private TaskProcessingTimeStats() { - } - - public static TaskProcessingTimeStats of(List tasks) { - TaskProcessingTimeStats s = new TaskProcessingTimeStats(); - - List processingTimesHrs = new ArrayList<>(tasks.size()); - - for (PipelineTask task : tasks) { - Date startProcessingTime = task.getStartProcessingTime(); - Date endProcessingTime = task.getEndProcessingTime(); - - if (startProcessingTime.getTime() > 0 - && startProcessingTime.getTime() < s.minStart.getTime()) { - s.minStart = startProcessingTime; - } - - if (endProcessingTime.getTime() > 0 - && endProcessingTime.getTime() > s.maxEnd.getTime()) { - s.maxEnd = endProcessingTime; - } - - processingTimesHrs - .add(DisplayModel.getProcessingHours(startProcessingTime, endProcessingTime)); - } - - s.totalElapsed = DisplayModel.getProcessingHours(s.minStart, s.maxEnd); - - DescriptiveStatistics stats = new DescriptiveStatistics(); - - for (Double d : processingTimesHrs) { - stats.addValue(d); - } - - s.count = tasks.size(); - s.sum = stats.getSum(); - s.min = stats.getMin(); - s.max = stats.getMax(); - s.mean = stats.getMean(); - s.stddev = stats.getStandardDeviation(); - - return s; - } - - public double getMin() { - return min; - } - - public double getMax() { - return max; - } - - public double getMean() { - return mean; - } - - public double getStddev() { - return stddev; - } - - public int getCount() { - return count; - } - - public Date getMinStart() { - return minStart; - } - - public Date getMaxEnd() { - return maxEnd; - } - - public double getTotalElapsed() { - return totalElapsed; - } - - public double getSum() { - return sum; - } -} diff --git a/src/main/java/gov/nasa/ziggy/util/ZiggyCollectionUtils.java b/src/main/java/gov/nasa/ziggy/util/ZiggyCollectionUtils.java index d64346c..51d6f6f 100644 --- a/src/main/java/gov/nasa/ziggy/util/ZiggyCollectionUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/ZiggyCollectionUtils.java @@ -8,7 +8,7 @@ import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; /** * Utilities for use with Java collections. diff --git a/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java b/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java index ff508b0..a9237f8 100644 --- a/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java @@ -65,7 +65,7 @@ public class ZiggyStringUtils { public static String[] convertStringArray(String input) { checkNotNull(input, "input"); - log.debug("convertStringArray got " + input); + log.debug("input={}", input); StringTokenizer st = new StringTokenizer(input, ","); String[] results = new String[st.countTokens()]; int i = 0; diff --git a/src/main/java/gov/nasa/ziggy/util/ZiggyUtils.java b/src/main/java/gov/nasa/ziggy/util/ZiggyUtils.java new file mode 100644 index 0000000..ce9b912 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/util/ZiggyUtils.java @@ -0,0 +1,73 @@ +package gov.nasa.ziggy.util; + +import static com.google.common.base.Preconditions.checkNotNull; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.module.PipelineException; + +/** + * Contains static methods that don't belong anywhere else. + * + * @author Bill Wohler + */ +public class ZiggyUtils { + private static final Logger log = LoggerFactory.getLogger(ZiggyUtils.class); + + @FunctionalInterface + public interface ThrowingSupplier { + T get() throws Exception; + } + + /** + * Tries to obtain a value from the provided supplier until a value is returned or the maximum + * number of tries is attempted. + *

+ * The {@code pauseMillis} should be chosen so that it is short enough to keep this call timely, + * but not too short to saturate the CPU. The value of {@code tries} should then be chosen so + * that this method gives up after the maximum time the supplier should return under normal + * circumstances. + * + * @param The type this method and the supplier return + * @param message The non-null log message with "(take 1/25)", for example, appended + * @param tries The maximum number of tries to attempt obtaining the value from the supplier + * @param pauseMillis The time to pause between each attempt to obtain the value from the + * supplier, in milliseconds + * @param supplier A non-null lambda that returns a value, which may or may not throw an + * exception + * @return The value that the supplier returns + * @throws PipelineException If the supplier never returns a value in the alloted time + */ + public static T tryPatiently(String message, int tries, long pauseMillis, + ThrowingSupplier supplier) { + + checkNotNull(message, "message"); + checkNotNull(supplier, "supplier"); + + for (int i = 1; i <= tries; i++) { + log.info("{} (take {}/{})", message, i, tries); + try { + return supplier.get(); + } catch (Exception e) { + if (i == tries) { + throw new PipelineException(e); + } + try { + Thread.sleep(pauseMillis); + } catch (InterruptedException interrupt) { + Thread.currentThread().interrupt(); + return null; + } + } + } + return null; + // In your house, I long to be + // Room by room, patiently + // I'll wait for you there + // Like a stone + // I'll wait for you there + // Alone + // Audioslave, "Like a Stone" + } +} diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/AlertLogDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/AlertLogDisplayModel.java index d156d0b..384f66c 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/AlertLogDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/AlertLogDisplayModel.java @@ -34,7 +34,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> alert.getSourceComponent(); - case 1 -> alert.getSourceTaskId(); + case 1 -> alert.getSourceTask(); case 2 -> alert.getSeverity(); case 3 -> alert.getMessage(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java index c05705d..8239038 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java @@ -86,6 +86,10 @@ public static double getProcessingMillis(Date start, Date end) { public static double getProcessingHours(Date start, Date end) { double processingMillis = getProcessingMillis(start, end); + return getProcessingHours(processingMillis); + } + + public static double getProcessingHours(double processingMillis) { return processingMillis / (1000.0 * 60.0 * 60.0); } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java index 736fc70..29709e5 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java @@ -57,7 +57,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { case 1 -> instance.getPipelineDefinition().getName() + (StringUtils.isBlank(instance.getName()) ? "" : ": " + instance.getName()); case 2 -> getStateString(instance.getState()); - case 3 -> instance.elapsedTime(); + case 3 -> instance.getCreated(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); }; } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java index 3d93ef3..298e6b7 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java @@ -1,15 +1,18 @@ package gov.nasa.ziggy.util.dispmod; +import java.util.Date; import java.util.HashMap; import java.util.LinkedList; import java.util.List; import java.util.Map; import java.util.Objects; +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; + import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.util.TaskProcessingTimeStats; import gov.nasa.ziggy.util.dispmod.PipelineStatsDisplayModel.ProcessingStatistics; /** @@ -25,32 +28,33 @@ public class PipelineStatsDisplayModel extends DisplayModel implements ModelContentClass { - private static final String[] COLUMN_NAMES = { "Module", "Status", "Count", "Sum (hrs)", - "Min (hrs)", "Max (hrs)", "Mean (hrs)", "Std (hrs)", "Start", "End", "Elapsed (hrs)" }; + private static final String[] COLUMN_NAMES = { "Module", "Start", "Status", "Count", + "Sum (hrs)", "Min (hrs)", "Max (hrs)", "Mean (hrs)", "Std (hrs)" }; private static final String ERROR = "ERROR"; private List stats = new LinkedList<>(); - public PipelineStatsDisplayModel(List tasks, List orderedModuleNames) { + public PipelineStatsDisplayModel(List tasks, + List orderedModuleNames) { update(tasks, orderedModuleNames); } - private void update(List tasks, List orderedModuleNames) { + private void update(List tasks, List orderedModuleNames) { stats = new LinkedList<>(); - Map>> moduleStats = new HashMap<>(); + Map>> moduleStats = new HashMap<>(); - for (PipelineTask task : tasks) { + for (PipelineTaskDisplayData task : tasks) { String moduleName = task.getModuleName(); - Map> moduleMap = moduleStats.get(moduleName); + Map> moduleMap = moduleStats.get(moduleName); if (moduleMap == null) { moduleMap = new HashMap<>(); moduleStats.put(moduleName, moduleMap); } - List tasksSubList = moduleMap.get(displayProcessingStep(task)); + List tasksSubList = moduleMap.get(displayProcessingStep(task)); if (tasksSubList == null) { tasksSubList = new LinkedList<>(); moduleMap.put(displayProcessingStep(task), tasksSubList); @@ -60,7 +64,7 @@ private void update(List tasks, List orderedModuleNames) { } for (String moduleName : orderedModuleNames) { - Map> moduleMap = moduleStats.get(moduleName); + Map> moduleMap = moduleStats.get(moduleName); updateStats(moduleName, moduleMap, ERROR); for (ProcessingStep processingStep : ProcessingStep.values()) { @@ -69,16 +73,16 @@ private void update(List tasks, List orderedModuleNames) { } } - private void updateStats(String moduleName, Map> moduleMap, - String displayProcessingStep) { - List tasksSubList = moduleMap.get(displayProcessingStep); + private void updateStats(String moduleName, + Map> moduleMap, String displayProcessingStep) { + List tasksSubList = moduleMap.get(displayProcessingStep); if (tasksSubList != null) { TaskProcessingTimeStats s = TaskProcessingTimeStats.of(tasksSubList); stats.add(new ProcessingStatistics(moduleName, displayProcessingStep, s)); } } - private String displayProcessingStep(PipelineTask task) { + private String displayProcessingStep(PipelineTaskDisplayData task) { return task.isError() ? ERROR : task.getProcessingStep().toString(); } @@ -99,16 +103,14 @@ public Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> statsForTaskType.getModuleName(); - case 1 -> statsForTaskType.getDisplayProcessingStep(); - case 2 -> s.getCount(); - case 3 -> formatDouble(s.getSum()); - case 4 -> formatDouble(s.getMin()); - case 5 -> formatDouble(s.getMax()); - case 6 -> formatDouble(s.getMean()); - case 7 -> formatDouble(s.getStddev()); - case 8 -> formatDate(s.getMinStart()); - case 9 -> formatDate(s.getMaxEnd()); - case 10 -> formatDouble(s.getTotalElapsed()); + case 1 -> formatDate(s.getMinStart()); + case 2 -> statsForTaskType.getDisplayProcessingStep(); + case 3 -> s.getCount(); + case 4 -> formatDouble(s.getSum()); + case 5 -> formatDouble(s.getMin()); + case 6 -> formatDouble(s.getMax()); + case 7 -> formatDouble(s.getMean()); + case 8 -> formatDouble(s.getStddev()); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); }; } @@ -167,4 +169,86 @@ public boolean equals(Object obj) { && displayProcessingStep == other.displayProcessingStep; } } + + /** + * Computes the following statistics based on the processing times for the specified list of + * pipeline tasks: + * + * 

+     * max
+     * min
+     * mean
+     * stddev
+     *
+ * + * @author Todd Klaus + */ + static class TaskProcessingTimeStats { + private int count; + private double sum; + private double min; + private double max; + private double mean; + private double stddev; + private Date minStart = new Date(); + + /** + * Private to prevent instantiation. Use static 'of' method to create instances. + */ + private TaskProcessingTimeStats() { + } + + public static TaskProcessingTimeStats of(List tasks) { + TaskProcessingTimeStats s = new TaskProcessingTimeStats(); + DescriptiveStatistics stats = new DescriptiveStatistics(); + + for (PipelineTaskDisplayData task : tasks) { + Date createdTime = task.getCreated(); + + if (createdTime.getTime() > 0 && createdTime.getTime() < s.minStart.getTime()) { + s.minStart = createdTime; + } + + stats.addValue( + DisplayModel.getProcessingHours(task.getExecutionClock().totalExecutionTime())); + } + + s.count = tasks.size(); + s.sum = stats.getSum(); + s.min = stats.getMin(); + s.max = stats.getMax(); + s.mean = stats.getMean(); + s.stddev = stats.getStandardDeviation(); + + return s; + } + + public double getMin() { + return min; + } + + public double getMax() { + return max; + } + + public double getMean() { + return mean; + } + + public double getStddev() { + return stddev; + } + + public int getCount() { + return count; + } + + public Date getMinStart() { + return minStart; + } + + public double getSum() { + return sum; + } + } } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/TaskMetricsDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/TaskMetricsDisplayModel.java index 8b5bcdf..0ce3010 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/TaskMetricsDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/TaskMetricsDisplayModel.java @@ -12,12 +12,12 @@ import gov.nasa.ziggy.metrics.TaskMetrics; import gov.nasa.ziggy.metrics.TimeAndPercentile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; /** * Display a table containing a row for each pipeline module and a column for each category defined - * in PipelineTask.summaryMetrics. + * in PipelineTask.pipelineTaskMetrics. *

* The cells of the table contain the total time spent on each category for all tasks for the * pipeline module and the percentage of the total processing time for all of the tasks. @@ -32,29 +32,30 @@ public class TaskMetricsDisplayModel extends DisplayModel { private boolean completedTasksOnly = false; - public TaskMetricsDisplayModel(List tasks, List orderedModuleNames) { + public TaskMetricsDisplayModel(List tasks, + List orderedModuleNames) { this(tasks, orderedModuleNames, true); } - public TaskMetricsDisplayModel(List tasks, List orderedModuleNames, - boolean completedTasksOnly) { + public TaskMetricsDisplayModel(List tasks, + List orderedModuleNames, boolean completedTasksOnly) { this.completedTasksOnly = completedTasksOnly; update(tasks, orderedModuleNames); } - private void update(List tasks, List orderedModuleNames) { + private void update(List tasks, List orderedModuleNames) { categorySummariesByModule = new LinkedList<>(); seenCategories = new ArrayList<>(); - Map> tasksByModule = new HashMap<>(); + Map> tasksByModule = new HashMap<>(); // partition the tasks by module - for (PipelineTask task : tasks) { + for (PipelineTaskDisplayData task : tasks) { if (!completedTasksOnly || task.getProcessingStep() == ProcessingStep.COMPLETE) { String moduleName = task.getModuleName(); - List taskListForModule = tasksByModule.get(moduleName); + List taskListForModule = tasksByModule.get(moduleName); if (taskListForModule == null) { taskListForModule = new LinkedList<>(); tasksByModule.put(moduleName, taskListForModule); @@ -66,7 +67,7 @@ private void update(List tasks, List orderedModuleNames) { // for each module, aggregate the summary metrics by category // and build a list of categories for (String moduleName : orderedModuleNames) { - List taskListForModule = tasksByModule.get(moduleName); + List taskListForModule = tasksByModule.get(moduleName); TaskMetrics taskMetrics = new TaskMetrics(taskListForModule); taskMetrics.calculate(); categorySummariesByModule.add(new ModuleTaskMetrics(moduleName, taskMetrics)); diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/TasksDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/TasksDisplayModel.java index 71bf8f9..584328d 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/TasksDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/TasksDisplayModel.java @@ -1,12 +1,11 @@ package gov.nasa.ziggy.util.dispmod; -import java.util.ArrayList; import java.util.LinkedList; import java.util.List; import javax.swing.JLabel; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.ui.util.HtmlBuilder; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; @@ -16,6 +15,7 @@ * on the console. * * @author Todd Klaus + * @author Bill Wohler */ public class TasksDisplayModel extends DisplayModel { private static final String[] COLUMN_NAMES = { "ID", "Module", "UOW", "Worker", "Status", @@ -28,26 +28,26 @@ public class TasksDisplayModel extends DisplayModel { ZiggySwingUtils.textWidth(new JLabel(), "Subtasks"), ZiggySwingUtils.textWidth(new JLabel(), "00:00:00") }; - private List tasks = new LinkedList<>(); - private final TaskCounts taskCounts; + private List tasks = new LinkedList<>(); + private TaskCounts taskCounts = new TaskCounts(); public TasksDisplayModel() { - taskCounts = new TaskCounts(); } - public TasksDisplayModel(List tasks) { - this.tasks = tasks; + public TasksDisplayModel(List tasks) { + update(tasks); + } - taskCounts = new TaskCounts(this.tasks); + public TasksDisplayModel(PipelineTaskDisplayData task) { + update(List.of(task)); } - public TasksDisplayModel(PipelineTask task) { - tasks = new ArrayList<>(100); - tasks.add(task); - taskCounts = new TaskCounts(tasks); + public void update(List tasks) { + this.tasks = tasks; + taskCounts = new TaskCounts(this.tasks); } - public PipelineTask getPipelineTaskForRow(int row) { + public PipelineTaskDisplayData getPipelineTaskForRow(int row) { return tasks.get(row); } @@ -63,17 +63,17 @@ public int getColumnCount() { @Override public Object getValueAt(int rowIndex, int columnIndex) { - PipelineTask task = tasks.get(rowIndex); + PipelineTaskDisplayData task = tasks.get(rowIndex); String value = switch (columnIndex) { - case 0 -> task.getId().toString(); + case 0 -> Long.toString(task.getPipelineTaskId()); case 1 -> task.getModuleName(); - case 2 -> task.uowTaskInstance().briefState(); + case 2 -> task.getBriefState(); case 3 -> task.getWorkerName(); case 4 -> task.getDisplayProcessingStep(); case 5 -> TaskCounts.subtaskCountsLabel(task.getCompletedSubtaskCount(), task.getTotalSubtaskCount(), task.getFailedSubtaskCount()); - case 6 -> task.elapsedTime(); + case 6 -> task.getExecutionClock().toString(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); }; diff --git a/src/main/java/gov/nasa/ziggy/util/io/ZiggyFileUtils.java b/src/main/java/gov/nasa/ziggy/util/io/ZiggyFileUtils.java index c5ac1c8..9ce4b31 100644 --- a/src/main/java/gov/nasa/ziggy/util/io/ZiggyFileUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/io/ZiggyFileUtils.java @@ -29,7 +29,7 @@ import java.util.stream.Collectors; import java.util.stream.Stream; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.io.FileUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -190,7 +190,7 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { @Override public FileVisitResult visitFileFailed(Path file, IOException exc) { - log.error("Unable to visit file " + file + " for checksum purposes.", exc); + log.error("Unable to visit file {} for checksum purposes", file, exc); return FileVisitResult.CONTINUE; } diff --git a/src/main/java/gov/nasa/ziggy/util/os/AbstractSysInfo.java b/src/main/java/gov/nasa/ziggy/util/os/AbstractSysInfo.java index 833a382..79d9b7a 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/AbstractSysInfo.java +++ b/src/main/java/gov/nasa/ziggy/util/os/AbstractSysInfo.java @@ -42,12 +42,12 @@ public void put(String key, String value) { protected void parse(Collection sysInfo) { for (String line : sysInfo) { - log.debug("line = " + line); + log.debug("line={}", line); if (line != null && line.trim().length() > 0) { String[] tokens = line.split(":"); if (tokens.length < 2) { - log.debug("ignoring line with two few tokens: " + line); + log.debug("Ignoring line with two few tokens: {}", line); continue; } put(tokens[0].trim(), tokens[1].trim()); diff --git a/src/main/java/gov/nasa/ziggy/util/os/LinuxProcInfo.java b/src/main/java/gov/nasa/ziggy/util/os/LinuxProcInfo.java index 3ccbd6c..db15d23 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/LinuxProcInfo.java +++ b/src/main/java/gov/nasa/ziggy/util/os/LinuxProcInfo.java @@ -66,7 +66,7 @@ public List getChildPids(String name) { if (ppid == currentPid && (name == null || name.equals(processName))) { // found a match - log.info("Found child process, pid=" + pid + ", name=" + processName); + log.info("Found child process, pid={}, name={}", pid, processName); childPids.add(pid); } } catch (Exception e) { diff --git a/src/main/java/gov/nasa/ziggy/util/os/MacOSXMemInfo.java b/src/main/java/gov/nasa/ziggy/util/os/MacOSXMemInfo.java index 9ba1208..0903cb6 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/MacOSXMemInfo.java +++ b/src/main/java/gov/nasa/ziggy/util/os/MacOSXMemInfo.java @@ -31,12 +31,12 @@ public MacOSXMemInfo() { @Override protected void parse(Collection topOutput) { for (String line : topOutput) { - log.debug("line = " + line); + log.debug("line={}", line); if (line != null && line.trim().length() > 0) { String[] tokens = line.trim().split("\\s+"); if (tokens.length < 2) { - log.debug("ignoring line with two few tokens: " + line); + log.debug("Ignoring line with two few tokens: {}", line); continue; } String field = tokens[0].trim().toLowerCase(); diff --git a/src/main/java/gov/nasa/ziggy/util/os/MacOSXProcInfo.java b/src/main/java/gov/nasa/ziggy/util/os/MacOSXProcInfo.java index 2af90ff..0b8afb5 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/MacOSXProcInfo.java +++ b/src/main/java/gov/nasa/ziggy/util/os/MacOSXProcInfo.java @@ -37,12 +37,12 @@ public MacOSXProcInfo() { @Override protected void parse(Collection commandOutput) { for (String line : commandOutput) { - log.debug("line = " + line); + log.debug("line={}", line); if (line != null && line.trim().length() > 0) { String[] tokens = line.trim().split("\\s+"); if (tokens.length < 2) { - log.debug("ignoring line with two few tokens: " + line); + log.debug("Ignoring line with two few tokens: {}", line); continue; } put("Pid", tokens[0]); @@ -64,19 +64,19 @@ public List getChildPids(String name) { List childPids = new LinkedList<>(); for (String line : commandOutput) { - log.debug("line = " + line); + log.debug("line={}", line); if (line != null && line.trim().length() > 0) { String[] tokens = line.trim().split("\\s+"); if (tokens.length < 2) { - log.debug("ignoring line with two few tokens: " + line); + log.debug("Ignoring line with two few tokens: {}", line); continue; } if (Long.parseLong(tokens[1]) == currentPid && (name == null || tokens[2].endsWith(name))) { // found a match long pid = Long.parseLong(tokens[0]); - log.info("Found child process, pid=" + pid + ", name=" + tokens[2]); + log.info("Found child process, pid={}, name={}", pid, tokens[2]); childPids.add(pid); } } diff --git a/src/main/java/gov/nasa/ziggy/util/os/OperatingSystemType.java b/src/main/java/gov/nasa/ziggy/util/os/OperatingSystemType.java index 2f8b4e3..a34d65e 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/OperatingSystemType.java +++ b/src/main/java/gov/nasa/ziggy/util/os/OperatingSystemType.java @@ -1,5 +1,6 @@ package gov.nasa.ziggy.util.os; +import static com.google.common.base.Preconditions.checkNotNull; import static gov.nasa.ziggy.services.config.PropertyName.OPERATING_SYSTEM; import static gov.nasa.ziggy.services.config.PropertyName.SUN_ARCH_DATA_MODEL; @@ -79,9 +80,7 @@ public ProcInfo getProcInfo() { } public static final OperatingSystemType byName(String name) { - if (name == null) { - throw new IllegalArgumentException("name cannot be null."); - } + checkNotNull(name, "name"); for (OperatingSystemType type : OperatingSystemType.values()) { if (type != OperatingSystemType.DEFAULT @@ -90,14 +89,12 @@ public static final OperatingSystemType byName(String name) { } } - log.warn(name + ": unrecognized operating system, using default type"); + log.warn("Unrecognized operating system {}, using default type", name); return OperatingSystemType.DEFAULT; } public static final OperatingSystemType byType(String name) { - if (name == null) { - throw new IllegalArgumentException("name cannot be null."); - } + checkNotNull(name, "name"); for (OperatingSystemType type : OperatingSystemType.values()) { if (type.toString().equalsIgnoreCase(name.trim().replace(' ', '_'))) { @@ -105,11 +102,11 @@ public static final OperatingSystemType byType(String name) { } } - log.warn(name + ": unrecognized operating system, using default type"); + log.warn("Unrecognized operating system {}, using default type", name); return OperatingSystemType.DEFAULT; } - public static final OperatingSystemType getInstance() { + public static final OperatingSystemType newInstance() { return OperatingSystemType .byType(ZiggyConfiguration.getInstance().getString(OPERATING_SYSTEM.property())); } diff --git a/src/main/java/gov/nasa/ziggy/util/os/ProcessUtils.java b/src/main/java/gov/nasa/ziggy/util/os/ProcessUtils.java index 806415b..922e9cf 100644 --- a/src/main/java/gov/nasa/ziggy/util/os/ProcessUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/os/ProcessUtils.java @@ -155,7 +155,7 @@ public static Process runJava(Class mainClass, List mainArgs) { String[] commandArray = new String[commandList.size()]; commandList.toArray(commandArray); try { - log.info("Executing java process with command line \"" + cmd + "\"."); + log.info("Executing Java process with command line {}", cmd); Process process = Runtime.getRuntime().exec(commandArray); BufferedReader errors = new BufferedReader( new InputStreamReader(process.getErrorStream(), ZiggyFileUtils.ZIGGY_CHARSET_NAME)); diff --git a/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java b/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java index a4dc40a..e90fbb7 100644 --- a/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java +++ b/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java @@ -34,6 +34,7 @@ package gov.nasa.ziggy.worker; +import java.util.List; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; @@ -41,12 +42,14 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.logging.TaskLog; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; import gov.nasa.ziggy.services.messages.HeartbeatMessage; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.services.messages.KilledTaskMessage; import gov.nasa.ziggy.services.messages.ShutdownMessage; +import gov.nasa.ziggy.services.messages.TaskHaltedMessage; import gov.nasa.ziggy.services.messaging.HeartbeatManager; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; @@ -71,6 +74,7 @@ * tasks. * * @author PT + * @author Bill Wohler */ public class PipelineWorker extends AbstractPipelineProcess { @@ -82,12 +86,14 @@ public class PipelineWorker extends AbstractPipelineProcess { public static final long FINAL_MESSAGE_LATCH_WAIT_MILLIS = 5000; private int workerId; - private long taskId; + private PipelineTask pipelineTask; private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); - public PipelineWorker(String name, int workerId) { + public PipelineWorker(String name, PipelineTask pipelineTask, int workerId) { super(name + " " + Integer.toString(workerId)); this.workerId = workerId; + this.pipelineTask = pipelineTask; } /** @@ -102,9 +108,10 @@ public static void main(String[] args) { RunMode runMode = RunMode.valueOf(args[2]); // Construct the WorkerProcess instance - PipelineWorker workerProcess = new PipelineWorker(NAME, workerId); + PipelineWorker workerProcess = new PipelineWorker(NAME, + new PipelineTaskOperations().pipelineTask(taskId), workerId); workerProcess.initialize(); - workerProcess.processTask(taskId, runMode); + workerProcess.processTask(runMode); log.info("Worker exiting with status 0"); SystemProxy.exit(0); } @@ -113,15 +120,14 @@ public static void main(String[] args) { * Main processing method. Instantiates the {@link ZiggyRmiClient} and {@link TaskExecutor} * instances, then starts the {@link TaskExecutor#executeTask()} method. */ - private void processTask(long taskId, RunMode runMode) { + private void processTask(RunMode runMode) { - this.taskId = taskId; TaskLog.endConsoleLogging(); - pipelineTaskOperations().incrementPipelineTaskLogIndex(taskId); + pipelineTaskDataOperations().incrementTaskLogIndex(pipelineTask); log.info("Process {} instance {} starting", NAME, workerId); // Initialize an instance of TaskExecutor - TaskExecutor taskRequestExecutor = new TaskExecutor(workerId, taskId, runMode); + TaskExecutor taskRequestExecutor = new TaskExecutor(workerId, pipelineTask, runMode); addProcessStatusReporter(taskRequestExecutor); // Initialize the ProcessHeartbeatManager for this process. @@ -135,6 +141,7 @@ private void processTask(long taskId, RunMode runMode) { // Note that we need to wait for the final status message to get sent // before we reset the ZiggyRmiClient. + log.debug("Executing worker shutdown hook"); CountDownLatch latch = new CountDownLatch(1); ZiggyMessenger.publish(taskRequestExecutor.statusMessage(true), latch); try { @@ -148,6 +155,11 @@ private void processTask(long taskId, RunMode runMode) { // Subscribe to messages as needed. subscribe(); + // Check for a halt request on the task. + if (pipelineTaskDataOperations().haltRequested(pipelineTask)) { + haltTask(new HaltTasksRequest(List.of(pipelineTask))); + } + // Start the TaskExecutor taskRequestExecutor.executeTask(); } @@ -173,36 +185,36 @@ private void subscribe() { killWorker(); }); - // When a kill-tasks request comes in, honor it if appropriate. - ZiggyMessenger.subscribe(KillTasksRequest.class, message -> { - killTasks(message); + // When a halt-tasks request comes in, honor it if appropriate. + ZiggyMessenger.subscribe(HaltTasksRequest.class, message -> { + haltTask(message); }); } /** - * Determines whether a {@link KillTasksRequest} wants to kill the task in this worker, and if + * Determines whether a {@link HaltTasksRequest} wants to halt the task in this worker, and if * so sends the desired confirmation method and exits. *

* Default axis (package-only) for unit tests. * * @param message */ - void killTasks(KillTasksRequest message) { - if (message.getTaskIds().contains(taskId)) { - sendKilledTaskMessage(message, taskId); + void haltTask(HaltTasksRequest message) { + if (message.getPipelineTasks().contains(pipelineTask)) { + sendTaskHaltedMessage(message, pipelineTask); killWorker(); } } /** - * Sends the {@link KilledTaskMessage} confirming that the task was in this worker and has been - * killed. + * Sends the {@link TaskHaltedMessage} confirming that the task was in this worker and has been + * halted. *

* Default access (package-only) for unit tests. */ - void sendKilledTaskMessage(KillTasksRequest message, long taskId) { + void sendTaskHaltedMessage(HaltTasksRequest message, PipelineTask pipelineTask) { CountDownLatch countDownLatch = new CountDownLatch(1); - ZiggyMessenger.publish(new KilledTaskMessage(message, taskId), countDownLatch); + ZiggyMessenger.publish(new TaskHaltedMessage(pipelineTask), countDownLatch); try { countDownLatch.await(FINAL_MESSAGE_LATCH_WAIT_MILLIS, TimeUnit.MILLISECONDS); } catch (InterruptedException e) { @@ -212,9 +224,9 @@ void sendKilledTaskMessage(KillTasksRequest message, long taskId) { /** * Kills the worker by calling {@link System#exit(int)}. This method is called when the worker - * receives a {@link KillTasksRequest} message from the supervisor, and the task processed in + * receives a {@link HaltTasksRequest} message from the supervisor, and the task processed in * this worker is one of the ones that is to be killed. Because the worker is a standalone - * process, and because each worker processes one and only one task, this approach (killing the + * process, and because each worker processes one and only one task, this approach (halting the * task by killing the worker) is the easiest and most robust way to ensure that the task is * stopped. *

@@ -225,11 +237,11 @@ void killWorker() { } /** For testing only. */ - void setTaskId(long taskId) { - this.taskId = taskId; - } - PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } } diff --git a/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java b/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java index 7a02656..7a0060c 100644 --- a/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java +++ b/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java @@ -3,7 +3,8 @@ import static gov.nasa.ziggy.services.process.AbstractPipelineProcess.getProcessInfo; import java.util.Date; -import java.util.List; +import java.util.Map; +import java.util.concurrent.CountDownLatch; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -16,19 +17,20 @@ import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.database.DatabaseService; +import gov.nasa.ziggy.services.messages.UpdateProcessingStepMessage; import gov.nasa.ziggy.services.messages.WorkerStatusMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; +import gov.nasa.ziggy.services.process.AbstractPipelineProcess; import gov.nasa.ziggy.services.process.StatusMessage; import gov.nasa.ziggy.services.process.StatusReporter; -import gov.nasa.ziggy.supervisor.TaskContext; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.BuildInfo; /** * Coordinates processing of inbound worker task request messages. Manages the database and @@ -36,8 +38,14 @@ * * @author Todd Klaus * @author PT + * @author Bill Wohler */ public class TaskExecutor implements StatusReporter { + + private enum TaskState { + IDLE, PROCESSING + } + private static final Logger log = LoggerFactory.getLogger(TaskExecutor.class); public static final String PIPELINE_MODULE_COMMIT_METRIC = "pipeline.module.commitTime"; @@ -53,19 +61,23 @@ public class TaskExecutor implements StatusReporter { private static PipelineTask workerTask; private final int workerNumber; - private String lastErrorMessage = ""; + private TaskState state = TaskState.IDLE; private boolean taskDone; - private TaskContext taskContext = new TaskContext(); - private long taskId; + private PipelineTask pipelineTask; + private PipelineModule currentPipelineModule; private RunMode runMode; + private long processingStartTimeMillis; + private Map taskMetrics; + private String lastErrorMessage = ""; + private CountDownLatch outgoingMessageLatch; + private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); - public TaskExecutor(int workerNumber, long taskId, RunMode runMode) { - this.taskId = taskId; + public TaskExecutor(int workerNumber, PipelineTask pipelineTask, RunMode runMode) { + this.pipelineTask = pipelineTask; this.workerNumber = workerNumber; this.runMode = runMode; - - taskContext.setTask(pipelineTaskOperations().pipelineTask(taskId)); } /** @@ -84,8 +96,8 @@ public void executeTask() { // all the foregoing catch and finally actions are complete. lastErrorMessage = e.getMessage(); - log.error("processMessage(): caught exception processing worker task request for " - + contextString(taskContext), e); + log.error("Caught exception processing worker task request for {}", + pipelineTask.toFullString(), e); CounterMetric.increment("pipeline.module.execFailCount"); @@ -101,83 +113,91 @@ public void executeTask() { private void executeTaskInternal() throws Exception { - taskContext.setState(TaskContext.TaskState.PROCESSING); - taskContext.setProcessingStartTimeMillis(System.currentTimeMillis()); + state = TaskState.PROCESSING; + processingStartTimeMillis = System.currentTimeMillis(); ZiggyMessenger.publish(statusMessage(false)); - log.info("Executing pre-processing for taskId={}", taskId); + log.info("Executing pre-processing for task {}", pipelineTask); preProcessing(); - log.info("Executing pre-processing for taskId={}...done", taskId); + log.info("Executing pre-processing for task {}...done", pipelineTask); try { Metric.enableThreadMetrics(); /* Invoke the module */ - log.info("Executing processTask for taskId={}", taskId); + log.info("Executing processTask for task {}", pipelineTask); taskDone = processTask(); if (taskDone) { - log.info("Executing processTask for taskId={}...done", taskId); + log.info("Executing processTask for task {}...done", pipelineTask); CounterMetric.increment("pipeline.module.execSuccessCount"); } else { log.info( - "Executing processTask for taskId={}...current step done (more steps remain)", - taskId); + "Executing processTask for task {}...current step done (more steps remain)", + pipelineTask); } } finally { - taskContext.setTaskMetrics(Metric.getThreadMetrics()); + taskMetrics = Metric.getThreadMetrics(); Metric.disableThreadMetrics(); } /* Update the task status */ - log.info("Executing post-processing for taskId={}", taskId); + log.info("Executing post-processing for task {}", pipelineTask); postProcessing(taskDone, true); - log.info("Executing post-processing for taskId={}...done", taskId); + log.info("Executing post-processing for task {}...done", pipelineTask); } @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) private void postProcessingAfterException() { - if (taskContext.getPipelineTaskId() != null) { + if (pipelineTask.getId() != null) { // could be null if it wasn't found in the db try { postProcessing(true, false); } catch (Exception e) { - log.error("Failed in postProcessing for: " + contextString(taskContext), e); + log.error("Failed in postProcessing for {}", pipelineTask.toFullString(), e); } } } /** * Update the PipelineTask in the db to reflect the fact that this worker is now processing the - * task. This is done with a local transaction (outside of the distributed transaction) and - * committed immediately so that the console will show updated status right away + * task. */ private boolean preProcessing() { - - PipelineTask pipelineTask = pipelineTaskOperations().addTaskExecutionLog(taskId, - workerNumber, taskContext.getProcessingStartTimeMillis()); + pipelineTaskDataOperations().addTaskExecutionLog(pipelineTask, + AbstractPipelineProcess.getProcessInfo().getHost(), workerNumber, + processingStartTimeMillis); + ProcessingStep processingStep = pipelineTaskDataOperations().processingStep(pipelineTask); // If the user requested that only the transition logic be re-run, or if the transition // logic previously failed, then we only need to re-run the transition logic - boolean transitionOnly = pipelineTask.getProcessingStep() == ProcessingStep.COMPLETE; + boolean transitionOnly = processingStep == ProcessingStep.COMPLETE; - if (transitionOnly) { - pipelineTaskOperations().merge(pipelineTask); - } else { - pipelineTask.setWorkerHost(getProcessInfo().getHost()); - pipelineTask.setWorkerThread(workerNumber); - pipelineTask.setSoftwareRevision( - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + if (!transitionOnly) { + pipelineTaskDataOperations().updateWorkerInfo(pipelineTask, getProcessInfo().getHost(), + workerNumber); + pipelineTaskDataOperations().updateZiggySoftwareRevision(pipelineTask, + BuildInfo.ziggyVersion()); + pipelineTaskDataOperations().updatePipelineSoftwareRevision(pipelineTask, + BuildInfo.pipelineVersion()); // If this is the first time we've called the module, set the processing step to the // first step that the module defines; otherwise, leave it alone and let the module pick // up where it left off. - if (pipelineTask.getProcessingStep().isInfrastructureStep()) { - ProcessingStep processingStep = pipelineTaskOperations() + if (processingStep.isInfrastructureStep()) { + ProcessingStep firstProcessingStep = pipelineTaskOperations() .moduleImplementation(pipelineTask, runMode) .processingSteps() .get(0); - pipelineTaskOperations().updateProcessingStep(pipelineTask, processingStep); + outgoingMessageLatch = new CountDownLatch(1); + ZiggyMessenger.publish( + new UpdateProcessingStepMessage(pipelineTask, firstProcessingStep), + outgoingMessageLatch); + try { + outgoingMessageLatch.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } } } return transitionOnly; @@ -195,7 +215,7 @@ private boolean processTask() throws Exception { try { return processTaskInternal(); } catch (Exception e) { - log.error("Failed in PipelineModule.processTask() for: " + contextString(taskContext)); + log.error("Failed to process {}", pipelineTask.toFullString()); CounterMetric.increment("pipeline.module.execFailCount"); throw e; } finally { @@ -208,26 +228,29 @@ private boolean processTask() throws Exception { private boolean processTaskInternal() throws Exception { - // Here's initialization using a builder pattern. - PipelineTask pipelineTask = pipelineTaskOperations().pipelineTask(taskId); - workerTask = pipelineTask; - String moduleExecMetricPrefix = "pipeline.module." + taskContext.getModule(); - - log.info("Processing {}", contextString(taskContext)); + log.info("Processing {}", pipelineTask.toFullString()); - PipelineModule currentPipelineModule = pipelineTaskOperations() - .moduleImplementation(pipelineTask, runMode); - taskContext.setPipelineModule(currentPipelineModule); + currentPipelineModule = pipelineTaskOperations().moduleImplementation(pipelineTask, + runMode); - String moduleSimpleName = taskContext.getPipelineModule().getClass().getSimpleName(); + String moduleSimpleName = currentPipelineModule.getClass().getSimpleName(); log.info("Calling {}.processTask()", moduleSimpleName); - pipelineModuleProcessTask(currentPipelineModule, moduleExecMetricPrefix); + pipelineModuleProcessTask(currentPipelineModule, + "pipeline.module." + pipelineTask.getModuleName()); log.info("Calling {}.processTask()...done", moduleSimpleName); if (isTaskDone()) { - pipelineTaskOperations().updateProcessingStep(taskId, ProcessingStep.COMPLETE); + outgoingMessageLatch = new CountDownLatch(1); + ZiggyMessenger.publish( + new UpdateProcessingStepMessage(pipelineTask, ProcessingStep.COMPLETE), + outgoingMessageLatch); + try { + outgoingMessageLatch.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } } return isTaskDone(); @@ -238,14 +261,11 @@ private void pipelineModuleProcessTask(PipelineModule currentPipelineModule, String moduleExecMetricPrefix) throws Exception { IntervalMetricKey key = IntervalMetric.start(); - long startTime = System.currentTimeMillis(); - try { - // Hand off control to the PipelineModule implementation + // Hand off control to the PipelineModule implementation. setTaskDone(currentPipelineModule.processTask()); } finally { IntervalMetric.stop(moduleExecMetricPrefix + ".processTask", key); - taskContext.setModuleExecTime(System.currentTimeMillis() - startTime); } } @@ -254,45 +274,39 @@ private void pipelineModuleProcessTask(PipelineModule currentPipelineModule, */ private void postProcessing(boolean done, boolean success) { - PipelineTask pipelineTask = pipelineTaskOperations().pipelineTask(taskId); - long processingEndTimeMillis = System.currentTimeMillis(); - long totalProcessingTimeMillis = processingEndTimeMillis - - taskContext.getProcessingStartTimeMillis(); + long totalProcessingTimeMillis = processingEndTimeMillis - processingStartTimeMillis; - log.info("Total processing time for this step (minutes): {}", + log.info("Processing for this step took {} minutes", totalProcessingTimeMillis / 1000.0 / 60.0); Date endProcessingTime = new Date(processingEndTimeMillis); // Update summary metrics. - taskContext.getPipelineModule() - .updateMetrics(pipelineTask, taskContext.getTaskMetrics(), totalProcessingTimeMillis); + currentPipelineModule.updateMetrics(pipelineTask, taskMetrics, totalProcessingTimeMillis); - List execLog = pipelineTaskOperations().execLogs(pipelineTask); - log.debug("execLog size={}", execLog.size()); - - if (execLog != null && !execLog.isEmpty()) { - TaskExecutionLog currentExecLog = execLog.get(execLog.size() - 1); - currentExecLog.setEndProcessingTime(endProcessingTime); - currentExecLog.setFinalProcessingStep(pipelineTask.getProcessingStep()); - } else { - log.warn("Task execution log is missing or empty for taskId={}", taskId); - } + pipelineTaskDataOperations().updateLastTaskExecutionLog(pipelineTask, endProcessingTime); if (!done) { return; } if (success) { - pipelineTaskOperations().updateProcessingStep(pipelineTask.getId(), - ProcessingStep.COMPLETE); + outgoingMessageLatch = new CountDownLatch(1); + ZiggyMessenger.publish( + new UpdateProcessingStepMessage(pipelineTask, ProcessingStep.COMPLETE), + outgoingMessageLatch); + try { + outgoingMessageLatch.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } } else { - pipelineTask.incrementFailureCount(); - pipelineTask = pipelineTaskOperations().taskErrored(pipelineTask); + pipelineTaskDataOperations().incrementFailureCount(pipelineTask); + pipelineTaskDataOperations().taskErrored(pipelineTask); AlertService alertService = AlertService.getInstance(); - alertService.generateAlert("PI(" + taskContext.getModule() + ")", pipelineTask.getId(), - AlertService.Severity.INFRASTRUCTURE, lastErrorMessage); + alertService.generateAlert("PI(" + pipelineTask.getModuleName() + ")", pipelineTask, + Severity.INFRASTRUCTURE, lastErrorMessage); } } @@ -304,28 +318,19 @@ private boolean isTaskDone() { return taskDone; } - private String contextString(TaskContext context) { - return "IID=" + context.getPipelineInstanceId() + ", TID=" + taskId + ", M=" - + context.getModule() + ", UOW=" + context.getModuleUow(); - } - @Override public StatusMessage reportCurrentStatus() { return statusMessage(false); } public StatusMessage statusMessage(boolean lastMessage) { - Long pipelineInstanceId = taskContext.getPipelineInstanceId(); - Long pipelineTaskId = taskContext.getPipelineTaskId(); + Long pipelineInstanceId = pipelineTask.getPipelineInstanceId(); String currentPipelineInstanceId = pipelineInstanceId == null ? "-" : "" + pipelineInstanceId.toString(); - String currentPipelineTaskId = pipelineTaskId == null ? "-" - : "" + pipelineTaskId.toString(); - WorkerStatusMessage message = new WorkerStatusMessage(workerNumber, - taskContext.getState().toString(), currentPipelineInstanceId, currentPipelineTaskId, - taskContext.getModule(), taskContext.getModuleUow(), - taskContext.getProcessingStartTimeMillis(), lastMessage); + WorkerStatusMessage message = new WorkerStatusMessage(workerNumber, state.toString(), + currentPipelineInstanceId, pipelineTask, pipelineTask.getModuleName(), + pipelineTask.getUnitOfWork().briefState(), processingStartTimeMillis, lastMessage); message.setSourceProcess(getProcessInfo()); return message; } @@ -337,4 +342,8 @@ public static PipelineTask getWorkerTask() { PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } } diff --git a/src/main/python/zigutils/stacktrace.py b/src/main/python/zigutils/stacktrace.py index 1d45658..72fb8ff 100644 --- a/src/main/python/zigutils/stacktrace.py +++ b/src/main/python/zigutils/stacktrace.py @@ -51,9 +51,9 @@ def __init__(self, traceback): self.line = traceback.tb_lineno ''' -Writes a Ziggy-formatted HDF5 file to the sub-task directory. The user supplies an optional +Writes a Ziggy-formatted HDF5 file to the subtask directory. The user supplies an optional "sequence number," which defaults to zero. The result is a file named "-error-.h5" -The worker later reads this and adds its contents to the log for the sub-task. +The worker later reads this and adds its contents to the log for the subtask. The standard pattern for writing a pipeline algorithm call in Python is the following: try: @@ -61,7 +61,7 @@ def __init__(self, traceback): except: ZiggyErrorWriter() -When multiple inputs per sub-task are used, the sequence number of the current input should be +When multiple inputs per subtask are used, the sequence number of the current input should be supplied as an argument to the ZiggyErrorWriter constructor. @author PT @@ -102,4 +102,4 @@ def __init__(self, seq_num=0): # write to HDF5 file hdf5_module_interface = Hdf5ModuleInterface() hdf5_module_interface.write_file(error_file_name, error_return) - \ No newline at end of file + diff --git a/src/test/java/gov/nasa/ziggy/FlakyTestCategory.java b/src/test/java/gov/nasa/ziggy/FlakyTestCategory.java new file mode 100644 index 0000000..2a20fe2 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/FlakyTestCategory.java @@ -0,0 +1,19 @@ +package gov.nasa.ziggy; + +/** + * Marker interface for tests that don't fail every time. To use mark the class or method with + * + * 

+ * {@literal @}Category (FlakyTestCategory.class)
+ *
+ * + * These tests are implicitly marked with {@link IntegrationTestCategory}. This means that tests + * that have this marker don't need an additional {@code IntegrationTestCategory} marker, nor does + * {@code FlakyTestCategory} have to be added to a list of categories in a build file that already + * excludes {@code IntegrationTestCategory}. However, this marker should probably be excluded by + * Gradle integration test tasks that include the {@code IntegrationTestCategory}. + * + * @author Bill Wohler + */ +public interface FlakyTestCategory extends IntegrationTestCategory { +} diff --git a/src/test/java/gov/nasa/ziggy/ReflectionEquals.java b/src/test/java/gov/nasa/ziggy/ReflectionEquals.java index 7827473..244a81d 100644 --- a/src/test/java/gov/nasa/ziggy/ReflectionEquals.java +++ b/src/test/java/gov/nasa/ziggy/ReflectionEquals.java @@ -79,7 +79,7 @@ private void compareObjects(String message, String fullyQualifiedFieldName, return; } - log.debug("comparing: " + message); + log.debug("Comparing {}", message); if (expectedObject == actualObject) { // same object, must be equal! @@ -233,7 +233,7 @@ private Object getFieldValue(Field field, Object object) throws IllegalAccessExc getterMethod.setAccessible(true); return getterMethod.invoke(object); } catch (Exception e) { - log.debug("No getter method found for field: " + field.getName()); + log.debug("No getter method found for field {}", field.getName()); } // then try isX @@ -242,7 +242,7 @@ private Object getFieldValue(Field field, Object object) throws IllegalAccessExc getterMethod.setAccessible(true); return getterMethod.invoke(object); } catch (Exception e) { - log.debug("No getter method found for field: " + field.getName()); + log.debug("No getter method found for field {}", field.getName()); } return field.get(object); diff --git a/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java b/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java index 1720928..67c11f4 100644 --- a/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java @@ -11,7 +11,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** * General utilities for unit and integration tests. @@ -33,15 +32,6 @@ public class ZiggyUnitTestUtils { // object. This is necessary because we need to be able to compare the objects with the // expected objects, and that comparison necessarily includes lazy-loaded bits. - // Initialization for database items that are specific to a particular execution of the - // pipeline: - // pipeline tasks, instances, instance nodes - public static void initializePipelineTask(PipelineTask pt) { - Hibernate.initialize(pt.getExecLog()); - Hibernate.initialize(pt.getSummaryMetrics()); - Hibernate.initialize(pt.getRemoteJobs()); - } - public static void initializePipelineInstance(PipelineInstance pipelineInstance) { Hibernate.initialize(pipelineInstance.getRootNodes()); Hibernate.initialize(pipelineInstance.getPipelineInstanceNodes()); diff --git a/src/test/java/gov/nasa/ziggy/crud/AbstractCrudTest.java b/src/test/java/gov/nasa/ziggy/crud/AbstractCrudTest.java new file mode 100644 index 0000000..324a805 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/crud/AbstractCrudTest.java @@ -0,0 +1,68 @@ +package gov.nasa.ziggy.crud; + +import static org.junit.Assert.assertEquals; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +import java.util.List; + +import org.junit.Before; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTask_; + +/** + * Unit tests for {@link AbstractCrud}. + * + * @author Bill Wohler + */ +public class AbstractCrudTest { + + @SuppressWarnings("unused") + private static final Logger log = LoggerFactory.getLogger(AbstractCrudTest.class); + + private TestCrud testCrud; + private ZiggyQuery ziggyQuery; + + @SuppressWarnings("unchecked") + @Before + public void setUp() { + testCrud = spy(new TestCrud()); + ziggyQuery = mock(ZiggyQuery.class); + } + + /** Shows that the list 1, 2, 3, 4, 5 is read in three chunks when maxExpressions == 2. */ + @Test + public void testChunkedQuery() { + doReturn(2).when(testCrud).maxExpressions(); + doReturn(List.of(1L, 2L), List.of(3L, 4L), List.of(5L)).when(testCrud).list(ziggyQuery); + + assertEquals(List.of(1L, 2L, 3L, 4L, 5L), + testCrud.chunkedPipelineTaskIds(List.of(1L, 2L, 3L, 4L, 5L))); + + verify(testCrud, times(3)).list(ziggyQuery); + } + + private class TestCrud extends AbstractCrud { + public List chunkedPipelineTaskIds(List pipelineTaskIds) { + doReturn(ziggyQuery).when(ziggyQuery).column(PipelineTask_.id); + doReturn(ziggyQuery).when(ziggyQuery).select(); + doReturn(ziggyQuery).when(ziggyQuery).in(anyList()); + + return chunkedQuery(pipelineTaskIds, + chunk -> list(ziggyQuery.column(PipelineTask_.id).select().in(chunk))); + } + + @Override + public Class componentClass() { + return PipelineTask.class; + } + } +} diff --git a/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java b/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java index 535de2c..cebffe5 100644 --- a/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java +++ b/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java @@ -28,7 +28,6 @@ import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; -import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyDirectoryRule; @@ -37,13 +36,16 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode_; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTask_; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData_; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; import gov.nasa.ziggy.pipeline.definition.UniqueNameVersionPipelineComponent_; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionCrud; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeCrud; import gov.nasa.ziggy.pipeline.definition.database.PipelineOperationsTestUtils; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskCrud; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataCrud; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.database.DatabaseOperations; import gov.nasa.ziggy.util.ZiggyStringUtils; @@ -58,10 +60,10 @@ public class ZiggyQueryTest { private final String HIBERNATE_LOG_FILE_NAME = "hibernate.log"; private final String APPENDER_NAME = "file-appender.log"; - private final String TABLE_NAME = "ziggy_PipelineTask"; + private final String TABLE_NAME = "ziggy_PipelineTaskData"; private final String DUMMY_TASK_NAME = "p1_0"; - private PipelineTaskCrud crud; + private PipelineTaskDataCrud crud; private Path logPath; private FileAppender hibernateLog; private TestOperations testOperations = new TestOperations(); @@ -97,7 +99,7 @@ public class ZiggyQueryTest { @Before public void setUp() { - crud = new PipelineTaskCrud(); + crud = new PipelineTaskDataCrud(); LoggerContext ctx = (LoggerContext) LogManager.getContext(false); Configuration c = ctx.getConfiguration(); @@ -132,8 +134,10 @@ public void shutDown() { @Test public void sqlRetrieveTest() throws IOException { - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertTrue(tasks.isEmpty()); // Read in the log file @@ -149,18 +153,15 @@ public void sqlRetrieveTest() throws IOException { int tableNameStart = queryString.indexOf("from " + TABLE_NAME + " " + DUMMY_TASK_NAME); assertTrue(tableNameStart > queryStart); - // Get the fields for a PipelineTask, removing any fields that are lazy initialized, as they - // don't appear in the query. - Field[] fields = PipelineTask.class.getDeclaredFields(); + // Get the fields for PipelineTaskData, removing any fields that are lazy initialized, + // transient, or embedded as they don't appear in the query. + Field[] fields = PipelineTaskData.class.getDeclaredFields(); List fieldNames = new ArrayList<>(); - Set lazyFieldNames = Set.of("log", "uowTaskParameters", "summaryMetrics", "execLog", - "producerTaskIds", "remoteJobs"); - Set transientFieldNames = Set.of("maxFailedSubtaskCount", "maxAutoResubmits", - "ERROR_PREFIX", "LOG_FILENAME_FORMAT"); + Set otherFieldNames = Set.of("executionClock", "pipelineTaskMetrics", + "taskExecutionLogs", "remoteJobs"); for (Field field : fields) { - if (!lazyFieldNames.contains(field.getName()) - && !transientFieldNames.contains(field.getName())) { + if (!otherFieldNames.contains(field.getName())) { fieldNames.add(field.getName()); } } @@ -169,12 +170,13 @@ public void sqlRetrieveTest() throws IOException { // statement. for (String fieldName : fieldNames) { String fullFieldName = DUMMY_TASK_NAME + "." + fieldName; - assertTrue(fieldName, queryString.indexOf(fullFieldName) > queryStart - && queryString.indexOf(fullFieldName) < tableNameStart); + int fullFieldNameIndex = queryString.indexOf(fullFieldName); + assertTrue(fieldName, + fullFieldNameIndex > queryStart && fullFieldNameIndex < tableNameStart); } - // None of the lazy fields should appear at all. - for (String fieldName : lazyFieldNames) { + // None of the lazy, transient, or embedded fields should appear at all. + for (String fieldName : otherFieldNames) { String fullFieldName = DUMMY_TASK_NAME + "." + fieldName; assertEquals(fieldName, -1, queryString.indexOf(fullFieldName)); } @@ -187,8 +189,10 @@ public void sqlRetrieveTest() throws IOException { public void testRetrieveAllInstances() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(4, tasks.size()); } @@ -198,11 +202,15 @@ public void testRetrieveAllInstances() { @Test public void testColumnCriterion() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.processingStep).in(ProcessingStep.COMPLETE); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.column(PipelineTaskData_.processingStep).in(ProcessingStep.COMPLETE); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(2, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(1L)); assertTrue(taskIds.contains(4L)); } @@ -214,11 +222,16 @@ public void testColumnCriterion() { @Test public void testWhereClause() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.where(query.getBuilder().between(query.getRoot().get(PipelineTask_.id), 2L, 3L)); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.where(query.getBuilder() + .between(query.getRoot().get(PipelineTaskData_.pipelineTaskId), 2L, 3L)); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(2, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(2L)); assertTrue(taskIds.contains(3L)); } @@ -230,11 +243,15 @@ public void testWhereClause() { @Test public void testGetInClauses() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.where(query.in(query.get(PipelineTask_.id), Set.of(1L, 2L))); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.where(query.in(query.get(PipelineTaskData_.pipelineTaskId), Set.of(1L, 2L))); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(2, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(1L)); assertTrue(taskIds.contains(2L)); } @@ -245,11 +262,15 @@ public void testGetInClauses() { @Test public void testScalarInClause() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.where(query.in(query.get(PipelineTask_.id), 1L)); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.where(query.in(query.get(PipelineTaskData_.pipelineTaskId), 1L)); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(1, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(1L)); } @@ -260,12 +281,16 @@ public void testScalarInClause() { @Test public void testColumnMultiValueCriterion() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.processingStep) + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.column(PipelineTaskData_.processingStep) .in(Set.of(ProcessingStep.COMPLETE, ProcessingStep.EXECUTING)); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(4, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(1L)); assertTrue(taskIds.contains(2L)); assertTrue(taskIds.contains(4L)); @@ -277,11 +302,15 @@ public void testColumnMultiValueCriterion() { @Test public void testColumnMultipleCriteria() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.id).in(Set.of(2L, 4L)).between(1L, 3L); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.column(PipelineTaskData_.pipelineTaskId).in(Set.of(2L, 4L)).between(1L, 3L); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(1, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(2L)); } @@ -291,14 +320,18 @@ public void testColumnMultipleCriteria() { @Test public void testCriteriaOnMultipleColumns() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.id) + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.column(PipelineTaskData_.pipelineTaskId) .in(Set.of(2L, 3L, 4L)) - .column(PipelineTask_.processingStep) + .column(PipelineTaskData_.processingStep) .in(ProcessingStep.COMPLETE); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(1, tasks.size()); - List taskIds = tasks.stream().map(PipelineTask::getId).collect(Collectors.toList()); + List taskIds = tasks.stream() + .map(t -> t.getPipelineTask().getId()) + .collect(Collectors.toList()); assertTrue(taskIds.contains(4L)); } @@ -308,14 +341,16 @@ public void testCriteriaOnMultipleColumns() { @Test public void testSort() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class); - query.column(PipelineTask_.id).descendingOrder(); - List tasks = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class); + query.column(PipelineTaskData_.pipelineTaskId).descendingOrder(); + List tasks = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(4, tasks.size()); - assertEquals(Long.valueOf(4L), tasks.get(0).getId()); - assertEquals(Long.valueOf(3L), tasks.get(1).getId()); - assertEquals(Long.valueOf(2L), tasks.get(2).getId()); - assertEquals(Long.valueOf(1L), tasks.get(3).getId()); + assertEquals(4L, tasks.get(0).getPipelineTask().getId().longValue()); + assertEquals(3L, tasks.get(1).getPipelineTask().getId().longValue()); + assertEquals(2L, tasks.get(2).getPipelineTask().getId().longValue()); + assertEquals(1L, tasks.get(3).getPipelineTask().getId().longValue()); } /** @@ -324,10 +359,10 @@ public void testSort() { @Test public void testSelectColumn() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.id).select(); - List ids = testOperations.performListQueryOnPipelineTaskTable(query); + query.column(PipelineTaskData_.pipelineTaskId).select(); + List ids = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(4, ids.size()); assertTrue(ids.contains(1L)); assertTrue(ids.contains(2L)); @@ -342,10 +377,10 @@ public void testSelectColumn() { @Test public void testSelectColumnWithColNameArg() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.select(PipelineTask_.id); - List ids = testOperations.performListQueryOnPipelineTaskTable(query); + query.select(PipelineTaskData_.pipelineTaskId); + List ids = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(4, ids.size()); assertTrue(ids.contains(1L)); assertTrue(ids.contains(2L)); @@ -360,10 +395,10 @@ public void testSelectColumnWithColNameArg() { @Test public void testSelectClause() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); query.select(query.getRoot().get("id")); - List ids = testOperations.performListQueryOnPipelineTaskTable(query); + List ids = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(4, ids.size()); assertTrue(ids.contains(1L)); assertTrue(ids.contains(2L)); @@ -377,10 +412,10 @@ public void testSelectClause() { @Test public void testNonDistinctSelection() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, - ProcessingStep.class); - query.column(PipelineTask_.processingStep).select(); - List steps = testOperations.performListQueryOnPipelineTaskTable(query); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class, ProcessingStep.class); + query.column(PipelineTaskData_.processingStep).select(); + List steps = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(4, steps.size()); assertTrue(steps.contains(ProcessingStep.COMPLETE)); assertTrue(steps.contains(ProcessingStep.EXECUTING)); @@ -393,11 +428,11 @@ public void testNonDistinctSelection() { @Test public void testDistinctSelection() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, - ProcessingStep.class); - query.column(PipelineTask_.processingStep).select(); + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class, ProcessingStep.class); + query.column(PipelineTaskData_.processingStep).select(); query.distinct(true); - List steps = testOperations.performListQueryOnPipelineTaskTable(query); + List steps = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(2, steps.size()); assertTrue(steps.contains(ProcessingStep.COMPLETE)); assertTrue(steps.contains(ProcessingStep.EXECUTING)); @@ -405,11 +440,11 @@ public void testDistinctSelection() { } private void assertErrorCount(int expectedErrors) { - ZiggyQuery errorCountQuery = crud.createZiggyQuery(PipelineTask.class, - Long.class); - errorCountQuery.column(PipelineTask_.error).in(true).count(); + ZiggyQuery errorCountQuery = crud + .createZiggyQuery(PipelineTaskData.class, Long.class); + errorCountQuery.column(PipelineTaskData_.error).in(true).count(); long errorCount = testOperations - .performUniqueResultQueryOnPipelineTaskTable(errorCountQuery); + .performUniqueResultQueryOnPipelineTaskDataTable(errorCountQuery); assertEquals(expectedErrors, errorCount); } @@ -419,9 +454,9 @@ private void assertErrorCount(int expectedErrors) { @Test public void testMax() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.id).max(); + query.column(PipelineTaskData_.pipelineTaskId).max(); Long maxId = testOperations.performUniqueResultQueryOnPipelineTaskTable(query); assertEquals(4L, maxId.longValue()); } @@ -432,9 +467,9 @@ public void testMax() { @Test public void testMin() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.id).min(); + query.column(PipelineTaskData_.pipelineTaskId).min(); Long minId = testOperations.performUniqueResultQueryOnPipelineTaskTable(query); assertEquals(1L, minId.longValue()); } @@ -445,9 +480,9 @@ public void testMin() { @Test public void testSum() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.id).sum(); + query.column(PipelineTaskData_.pipelineTaskId).sum(); Long idSum = testOperations.performUniqueResultQueryOnPipelineTaskTable(query); assertEquals(10L, idSum.longValue()); } @@ -458,12 +493,12 @@ public void testSum() { @Test public void testMultiSelect() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Object[].class); - query.column(PipelineTask_.id).select(); - query.column(PipelineTask_.processingStep).select(); - query.column(PipelineTask_.id).descendingOrder(); - List results = testOperations.performListQueryOnPipelineTaskTable(query); + query.column(PipelineTaskData_.pipelineTaskId).select(); + query.column(PipelineTaskData_.processingStep).select(); + query.column(PipelineTaskData_.pipelineTaskId).descendingOrder(); + List results = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(4, results.size()); assertEquals(4L, ((Long) results.get(0)[0]).longValue()); assertEquals(ProcessingStep.COMPLETE, results.get(0)[1]); @@ -474,10 +509,10 @@ public void testMultiSelect() { assertEquals(1L, ((Long) results.get(3)[0]).longValue()); assertEquals(ProcessingStep.COMPLETE, results.get(3)[1]); - query = crud.createZiggyQuery(PipelineTask.class, Object[].class); - query.column(PipelineTask_.id).select(); - query.column(PipelineTask_.error).in(true); - results = testOperations.performListQueryOnPipelineTaskTable(query); + query = crud.createZiggyQuery(PipelineTaskData.class, Object[].class); + query.column(PipelineTaskData_.pipelineTaskId).select(); + query.column(PipelineTaskData_.error).in(true); + results = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(1, results.size()); assertEquals(3L, ((Long) results.get(0)[0]).longValue()); } @@ -488,10 +523,10 @@ public void testMultiSelect() { @Test public void testMinMax() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Object[].class); - query.column(PipelineTask_.id).minMax(); - List results = testOperations.performListQueryOnPipelineTaskTable(query); + query.column(PipelineTaskData_.pipelineTaskId).minMax(); + List results = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(1, results.size()); assertEquals(1L, ((Long) results.get(0)[0]).longValue()); assertEquals(4L, ((Long) results.get(0)[1]).longValue()); @@ -503,11 +538,11 @@ public void testMinMax() { @Test public void testCombineWhereSelect() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.processingStep).in(ProcessingStep.COMPLETE); - query.column(PipelineTask_.id).select(); - List results = testOperations.performListQueryOnPipelineTaskTable(query); + query.column(PipelineTaskData_.processingStep).in(ProcessingStep.COMPLETE); + query.column(PipelineTaskData_.pipelineTaskId).select(); + List results = testOperations.performListQueryOnPipelineTaskDataTable(query); assertEquals(2, results.size()); assertTrue(results.contains(1L)); assertTrue(results.contains(4L)); @@ -516,12 +551,13 @@ public void testCombineWhereSelect() { @Test public void testWhereScalarInScalar() { populatePipelineTasks(); - ZiggyQuery query = crud - .createZiggyQuery(PipelineTask.class, ProcessingStep.class) - .column(PipelineTask_.processingStep) + ZiggyQuery query = crud + .createZiggyQuery(PipelineTaskData.class, ProcessingStep.class) + .column(PipelineTaskData_.processingStep) .select() .in(ProcessingStep.COMPLETE); - List results = testOperations.performListQueryOnPipelineTaskTable(query); + List results = testOperations + .performListQueryOnPipelineTaskDataTable(query); assertEquals(2, results.size()); assertTrue(results.contains(ProcessingStep.COMPLETE)); } @@ -536,43 +572,15 @@ public void testWhereCollectionInScalar() { .in(pipelineTasks.get(0)); } - /** - * Tests that the {#link {@link ZiggyQuery#chunkedIn(java.util.Collection)}} works as expected. - */ - @Test - public void testChunkedIn() { - populatePipelineTasks(); - final ZiggyQuery query = Mockito - .spy(crud.createZiggyQuery(PipelineTask.class, Long.class)); - Mockito.doReturn(2).when(query).maxExpressions(); - query.column(PipelineTask_.id).select().chunkedIn(Set.of(1L, 2L, 3L, 4L)); - List results = testOperations.performListQueryOnPipelineTaskTable(query); - assertEquals(4, results.size()); - assertTrue(results.contains(1L)); - assertTrue(results.contains(2L)); - assertTrue(results.contains(3L)); - assertTrue(results.contains(4L)); - List> queryChunks = query.queryChunks(); - assertEquals(2, queryChunks.size()); - assertEquals(2, queryChunks.get(0).size()); - assertEquals(2, queryChunks.get(1).size()); - List allQueryChunks = new ArrayList<>(queryChunks.get(0)); - allQueryChunks.addAll(queryChunks.get(1)); - assertTrue(allQueryChunks.contains(1L)); - assertTrue(allQueryChunks.contains(2L)); - assertTrue(allQueryChunks.contains(3L)); - assertTrue(allQueryChunks.contains(4L)); - } - /** * Exercises the {@link ZiggyQuery#count()} method. */ @Test public void testCount() { populatePipelineTasks(); - ZiggyQuery query = crud.createZiggyQuery(PipelineTask.class, + ZiggyQuery query = crud.createZiggyQuery(PipelineTaskData.class, Long.class); - query.column(PipelineTask_.processingStep).in(ProcessingStep.COMPLETE); + query.column(PipelineTaskData_.processingStep).in(ProcessingStep.COMPLETE); query.count(); Long resultCount = testOperations.performUniqueResultQueryOnPipelineTaskTable(query); assertEquals(2L, resultCount.longValue()); @@ -665,19 +673,14 @@ private void populatePipelineTasks() { PipelineInstanceNode pipelineInstanceNode = testOperations .merge(new PipelineInstanceNode()); - PipelineTask pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.COMPLETE); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.EXECUTING); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.EXECUTING); - pipelineTask.setError(true); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.COMPLETE); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.COMPLETE, false, + pipelineInstanceNode); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.EXECUTING, false, + pipelineInstanceNode); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.EXECUTING, true, + pipelineInstanceNode); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.COMPLETE, false, + pipelineInstanceNode); testOperations.merge(pipelineInstanceNode); } @@ -688,19 +691,15 @@ private void populatePipelineTasksWithTwoInstances() { PipelineInstanceNode pipelineInstanceNode2 = testOperations .merge(new PipelineInstanceNode()); - PipelineTask pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.COMPLETE); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode1); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.EXECUTING); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode1); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.EXECUTING); - pipelineTask.setError(true); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode2); - pipelineTask = new PipelineTask(); - pipelineTask.setProcessingStep(ProcessingStep.COMPLETE); - testOperations.persistPipelineTask(pipelineTask, pipelineInstanceNode2); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.COMPLETE, false, + pipelineInstanceNode1); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.EXECUTING, false, + pipelineInstanceNode1); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.EXECUTING, true, + pipelineInstanceNode2); + testOperations.persistPipelineTask(new PipelineTask(), ProcessingStep.COMPLETE, false, + pipelineInstanceNode2); + testOperations.merge(pipelineInstanceNode1); testOperations.merge(pipelineInstanceNode2); } @@ -711,11 +710,21 @@ public List performListQueryOnPipelineTaskTable(ZiggyQuery new PipelineTaskCrud().list(query)); } + public List performListQueryOnPipelineTaskDataTable( + ZiggyQuery query) { + return performTransaction(() -> new PipelineTaskDataCrud().list(query)); + } + public T performUniqueResultQueryOnPipelineTaskTable( - ZiggyQuery query) { + ZiggyQuery query) { return performTransaction(() -> new PipelineTaskCrud().uniqueResult(query)); } + public T performUniqueResultQueryOnPipelineTaskDataTable( + ZiggyQuery query) { + return performTransaction(() -> new PipelineTaskDataCrud().uniqueResult(query)); + } + public T performUniqueResultQueryOnPipelineInstanceNodeTable( ZiggyQuery query) { return performTransaction(() -> new PipelineInstanceNodeCrud().uniqueResult(query)); @@ -731,12 +740,19 @@ public T performUniqueResultQueryOnPipelineDefinitionTable( return performTransaction(() -> new PipelineDefinitionCrud().uniqueResult(query)); } - public void persistPipelineTask(PipelineTask pipelineTask, + protected PipelineTask persistPipelineTask(PipelineTask pipelineTask, + ProcessingStep processingStep, boolean error, PipelineInstanceNode pipelineInstanceNode) { - PipelineTask mergedTask = performTransaction( - () -> new PipelineTaskCrud().merge(pipelineTask)); + + PipelineTask mergedTask = performTransaction(() -> { + PipelineTask task = new PipelineTaskCrud().merge(pipelineTask); + new PipelineTaskDataOperations().createPipelineTaskData(task, processingStep); + new PipelineTaskDataOperations().setError(task, error); + return task; + }); pipelineInstanceNode.addPipelineTask(mergedTask); pipelineTasks.add(mergedTask); + return mergedTask; } public PipelineInstanceNode merge(PipelineInstanceNode pipelineInstanceNode) { diff --git a/src/test/java/gov/nasa/ziggy/data/accounting/DataAccountabilityReportTest.java b/src/test/java/gov/nasa/ziggy/data/accounting/DataAccountabilityReportTest.java index b080fd8..96d2a9b 100644 --- a/src/test/java/gov/nasa/ziggy/data/accounting/DataAccountabilityReportTest.java +++ b/src/test/java/gov/nasa/ziggy/data/accounting/DataAccountabilityReportTest.java @@ -17,9 +17,9 @@ import gov.nasa.ziggy.ReflectionEquals; import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.data.management.DatastoreProducerConsumerOperations; +import gov.nasa.ziggy.pipeline.definition.Parameter; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; -import gov.nasa.ziggy.pipeline.definition.Parameter; import gov.nasa.ziggy.uow.UnitOfWork; /** @@ -261,10 +261,10 @@ public PipelineTask pipelineTask(long pipelineTaskId) { PipelineTask task = Mockito.spy(PipelineTask.class); Mockito.doReturn(pipelineTaskId).when(task).getId(); - UnitOfWork uowt = new UnitOfWork(); - uowt.addParameter(new Parameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.addParameter(new Parameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, "Stub taskId = " + pipelineTaskId, ZiggyDataType.ZIGGY_STRING)); - task.setUowTaskParameters(uowt.getParameters()); + Mockito.doReturn(unitOfWork).when(task).getUnitOfWork(); return task; } diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerFullLocationTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerFullLocationTest.java index 3ffb6df..aff880f 100644 --- a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerFullLocationTest.java +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerFullLocationTest.java @@ -210,7 +210,7 @@ public void setUp() throws IOException { List uows = PipelineExecutor.generateUnitsOfWork(uowGenerator, pipelineInstanceNode); - Mockito.doReturn(uows.get(0)).when(pipelineTask).uowTaskInstance(); + Mockito.doReturn(uows.get(0)).when(pipelineTask).getUnitOfWork(); } /** Constructs a collection of zero-length files in the datastore. */ diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java index 93ce814..13fcf67 100644 --- a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java @@ -4,6 +4,9 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; import java.io.IOException; import java.nio.file.Files; @@ -36,11 +39,11 @@ import gov.nasa.ziggy.pipeline.definition.ModelType; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionNodeOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; -import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; @@ -83,23 +86,20 @@ public class DatastoreFileManagerTest { private ModelRegistry modelRegistry; private ModelMetadata modelMetadata; private Map regexpValueByName = new HashMap<>(); - private PipelineTaskOperations pipelineTaskOperations = Mockito - .mock(PipelineTaskOperations.class); - private PipelineDefinitionOperations pipelineDefinitionOperations = Mockito - .mock(PipelineDefinitionOperations.class); - private DatastoreProducerConsumerOperations datastoreProducerConsumerOperations = Mockito - .mock(DatastoreProducerConsumerOperations.class); - private PipelineDefinitionNodeOperations pipelineDefinitionNodeOperations = Mockito - .mock(PipelineDefinitionNodeOperations.class); + private PipelineTaskOperations pipelineTaskOperations = mock(PipelineTaskOperations.class); + private PipelineDefinitionOperations pipelineDefinitionOperations = mock( + PipelineDefinitionOperations.class); + private DatastoreProducerConsumerOperations datastoreProducerConsumerOperations = mock( + DatastoreProducerConsumerOperations.class); + private PipelineDefinitionNodeOperations pipelineDefinitionNodeOperations = mock( + PipelineDefinitionNodeOperations.class); @Before public void setUp() throws IOException { taskDirectory = DirectoryProperties.taskDataDir().toAbsolutePath(); - pipelineTask = Mockito.mock(PipelineTask.class); + pipelineTask = mock(PipelineTask.class); datastoreFileManager = Mockito.spy(new DatastoreFileManager(pipelineTask, taskDirectory)); - Mockito.doReturn(Mockito.mock(AlertService.class)) - .when(datastoreFileManager) - .alertService(); + doReturn(mock(AlertService.class)).when(datastoreFileManager).alertService(); // Create datastore directories. DatastoreTestUtils.createDatastoreDirectories(); @@ -115,8 +115,7 @@ public void setUp() throws IOException { .get("uncalibrated collateral pixel values"); allFilesAllSubtasksDataFileType = Mockito .spy(dataFileTypes.get("calibrated science pixel values")); - Mockito.when(allFilesAllSubtasksDataFileType.isIncludeAllFilesInAllSubtasks()) - .thenReturn(true); + when(allFilesAllSubtasksDataFileType.isIncludeAllFilesInAllSubtasks()).thenReturn(true); calibratedCollateralPixelDataFileType = dataFileTypes .get("calibrated collateral pixel values"); @@ -124,20 +123,15 @@ public void setUp() throws IOException { regexpsByName = DatastoreTestUtils.regexpsByName(); datastoreWalker = new DatastoreWalker(regexpsByName, DatastoreTestUtils.datastoreNodesByFullPath()); - Mockito.doReturn(datastoreWalker).when(datastoreFileManager).datastoreWalker(); - Mockito.doReturn(pipelineTaskOperations) - .when(datastoreFileManager) - .pipelineTaskOperations(); - Mockito.doReturn(pipelineDefinitionOperations) - .when(datastoreFileManager) + doReturn(datastoreWalker).when(datastoreFileManager).datastoreWalker(); + doReturn(pipelineTaskOperations).when(datastoreFileManager).pipelineTaskOperations(); + doReturn(pipelineDefinitionOperations).when(datastoreFileManager) .pipelineDefinitionOperations(); - Mockito.when(pipelineDefinitionOperations.processingMode(ArgumentMatchers.anyString())) + when(pipelineDefinitionOperations.processingMode(ArgumentMatchers.anyString())) .thenReturn(ProcessingMode.PROCESS_ALL); - Mockito.doReturn(datastoreProducerConsumerOperations) - .when(datastoreFileManager) + doReturn(datastoreProducerConsumerOperations).when(datastoreFileManager) .datastoreProducerConsumerOperations(); - Mockito.doReturn(pipelineDefinitionNodeOperations) - .when(datastoreFileManager) + doReturn(pipelineDefinitionNodeOperations).when(datastoreFileManager) .pipelineDefinitionNodeOperations(); // Construct the Map from regexp name to value. Note that we need to include the pixel type @@ -167,54 +161,44 @@ public void setUp() throws IOException { Files.createFile(modelMetadata.datastoreModelPath()); // Set up the pipeline task. - pipelineInstanceNode = Mockito.mock(PipelineInstanceNode.class); - pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); - Mockito.when(pipelineDefinitionNode.getPipelineName()).thenReturn("test pipeline"); - Mockito.when(pipelineTask.getModuleName()).thenReturn("test module"); - Mockito - .when(pipelineTaskOperations - .pipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn(pipelineDefinitionNode); - Mockito - .when(pipelineDefinitionNodeOperations - .inputDataFileTypes(ArgumentMatchers.any(PipelineDefinitionNode.class))) - .thenReturn(Set.of(uncalibratedSciencePixelDataFileType, - uncalibratedCollateralPixelDataFileType, allFilesAllSubtasksDataFileType)); - Mockito.when(pipelineTaskOperations.modelTypes(ArgumentMatchers.any(PipelineTask.class))) + pipelineInstanceNode = mock(PipelineInstanceNode.class); + pipelineDefinitionNode = mock(PipelineDefinitionNode.class); + when(pipelineDefinitionNode.getPipelineName()).thenReturn("test pipeline"); + when(pipelineTask.getModuleName()).thenReturn("test module"); + when( + pipelineTaskOperations.pipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) + .thenReturn(pipelineDefinitionNode); + when(pipelineDefinitionNodeOperations + .inputDataFileTypes(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(Set.of(uncalibratedSciencePixelDataFileType, + uncalibratedCollateralPixelDataFileType, allFilesAllSubtasksDataFileType)); + when(pipelineTaskOperations.modelTypes(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(Set.of(modelType)); - Mockito - .when(pipelineTaskOperations - .pipelineDefinitionName(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn("test pipeline"); - Mockito - .when( - pipelineTaskOperations.inputDataFileTypes(ArgumentMatchers.any(PipelineTask.class))) + when( + pipelineTaskOperations.pipelineDefinitionName(ArgumentMatchers.any(PipelineTask.class))) + .thenReturn("test pipeline"); + when(pipelineTaskOperations.inputDataFileTypes(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(Set.of(uncalibratedSciencePixelDataFileType, uncalibratedCollateralPixelDataFileType, allFilesAllSubtasksDataFileType)); - Mockito - .when(pipelineTaskOperations - .outputDataFileTypes(ArgumentMatchers.any(PipelineTask.class))) + when(pipelineTaskOperations.outputDataFileTypes(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); - modelRegistry = Mockito.mock(ModelRegistry.class); - Mockito.when(modelRegistry.getModels()).thenReturn(Map.of(modelType, modelMetadata)); - Mockito.when(pipelineTaskOperations.modelRegistry(ArgumentMatchers.any(PipelineTask.class))) + modelRegistry = mock(ModelRegistry.class); + when(modelRegistry.getModels()).thenReturn(Map.of(modelType, modelMetadata)); + when(pipelineTaskOperations.modelRegistry(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(modelRegistry); // Construct the UOW. DatastoreDirectoryUnitOfWorkGenerator uowGenerator = Mockito .spy(DatastoreDirectoryUnitOfWorkGenerator.class); - Mockito.doReturn(datastoreWalker).when(uowGenerator).datastoreWalker(); - Mockito.doReturn(pipelineDefinitionNodeOperations) - .when(uowGenerator) + doReturn(datastoreWalker).when(uowGenerator).datastoreWalker(); + doReturn(pipelineDefinitionNodeOperations).when(uowGenerator) .pipelineDefinitionNodeOperations(); - Mockito.doReturn(pipelineDefinitionNode) - .when(pipelineInstanceNode) - .getPipelineDefinitionNode(); + doReturn(pipelineDefinitionNode).when(pipelineInstanceNode).getPipelineDefinitionNode(); List uows = PipelineExecutor.generateUnitsOfWork(uowGenerator, pipelineInstanceNode); - Mockito.doReturn(uows.get(0)).when(pipelineTask).uowTaskInstance(); + doReturn(uows.get(0)).when(pipelineTask).getUnitOfWork(); } /** Constructs a collection of zero-length files in the datastore. */ @@ -309,7 +293,7 @@ private void checkForFiles(String baseName, Set subtaskFiles) { /** Tests that filesForSubtasks acts as expected for a single-subtask use case. */ @Test public void testFilesForSubtasksSingleSubtask() { - Mockito.when(pipelineDefinitionNode.getSingleSubtask()).thenReturn(true); + when(pipelineDefinitionNode.getSingleSubtask()).thenReturn(true); Set subtaskDefinitions = datastoreFileManager.subtaskDefinitions(); assertEquals(1, subtaskDefinitions.size()); SubtaskDefinition subtaskDefinition = subtaskDefinitions.iterator().next(); @@ -391,7 +375,7 @@ public void testCopyDatastoreFilesToTaskDirectory() { @Test public void testCopyTaskDirectoryFilesToDatastore() throws IOException { createOutputFiles(); - Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) + when(pipelineDefinitionNode.getOutputDataFileTypes()) .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); Set copiedFiles = datastoreFileManager.copyTaskDirectoryFilesToDatastore(); Path datastorePath = DirectoryProperties.datastoreRootDir() @@ -425,7 +409,7 @@ public void testInputFilesByOutputStatus() throws IOException { createOutputFiles(); createInputFiles(); setAlgorithmStateFiles(); - Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) + when(pipelineDefinitionNode.getOutputDataFileTypes()) .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); Files.delete(taskDirectory.resolve(SubtaskUtils.subtaskDirName(SUBTASK_DIR_COUNT - 1)) .resolve("outputs-file-" + (SUBTASK_DIR_COUNT - 1) + ".nc")); @@ -493,21 +477,20 @@ private void setAlgorithmStateFiles() { for (int subtaskIndex = 0; subtaskIndex < SUBTASK_DIR_COUNT - 1; subtaskIndex++) { AlgorithmStateFiles stateFile = new AlgorithmStateFiles( SubtaskUtils.subtaskDirectory(taskDirectory, subtaskIndex).toFile()); - stateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + stateFile.updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); stateFile.setOutputsFlag(); } new AlgorithmStateFiles( SubtaskUtils.subtaskDirectory(taskDirectory, SUBTASK_DIR_COUNT - 1).toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); } @Test public void testSingleSubtaskNoPerSubtaskFiles() { - Mockito - .when(pipelineDefinitionNodeOperations - .inputDataFileTypes(ArgumentMatchers.any(PipelineDefinitionNode.class))) - .thenReturn(Set.of(allFilesAllSubtasksDataFileType)); - Mockito.doReturn(true).when(pipelineDefinitionNode).getSingleSubtask(); + when(pipelineDefinitionNodeOperations + .inputDataFileTypes(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(Set.of(allFilesAllSubtasksDataFileType)); + doReturn(true).when(pipelineDefinitionNode).getSingleSubtask(); Set subtaskDefinitions = datastoreFileManager.subtaskDefinitions(); assertEquals(1, subtaskDefinitions.size()); SubtaskDefinition subtaskDefinition = subtaskDefinitions.iterator().next(); @@ -569,7 +552,7 @@ public void testFilterOutFilesAlreadyProcessed() { @Test public void testFilteringForSingleSubtask() { configureForFilteringTest(); - Mockito.when(pipelineDefinitionNode.getSingleSubtask()).thenReturn(true); + when(pipelineDefinitionNode.getSingleSubtask()).thenReturn(true); Set subtaskDefinitions = datastoreFileManager .subtaskDefinitions(pipelineDefinitionNode); @@ -585,10 +568,9 @@ public void testFilteringForSingleSubtask() { @Test public void testFilteringNoPriorProcessingDetected() { configureForFilteringTest(); - Mockito - .when(pipelineTaskOperations - .taskIdsForPipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn(new ArrayList<>()); + when(pipelineTaskOperations + .tasksForPipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) + .thenReturn(new ArrayList<>()); Set subtaskDefinitions = datastoreFileManager.subtaskDefinitions(); assertEquals(7, subtaskDefinitions.size()); } @@ -596,28 +578,30 @@ public void testFilteringNoPriorProcessingDetected() { private void configureForFilteringTest() { // Request processing of only new data. - Mockito.when(pipelineDefinitionOperations.processingMode(ArgumentMatchers.anyString())) + when(pipelineDefinitionOperations.processingMode(ArgumentMatchers.anyString())) .thenReturn(ProcessingMode.PROCESS_NEW); Set scienceDatastoreFilenames = producerConsumerTableFilenames("science"); Set collateralDatastoreFilenames = producerConsumerTableFilenames("collateral"); // Set up the retrieval of earlier consumer task IDs from the database. - Mockito - .when(pipelineTaskOperations - .taskIdsForPipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn(List.of(30L, 40L)) - .thenReturn(List.of(30L, 35L)); + PipelineTask pipelineTask30 = mock(PipelineTask.class); + when(pipelineTask30.getId()).thenReturn(30L); + PipelineTask pipelineTask35 = mock(PipelineTask.class); + when(pipelineTask35.getId()).thenReturn(35L); + PipelineTask pipelineTask40 = mock(PipelineTask.class); + when(pipelineTask40.getId()).thenReturn(40L); + + when(pipelineTaskOperations + .tasksForPipelineDefinitionNode(ArgumentMatchers.any(PipelineTask.class))) + .thenReturn(List.of(pipelineTask30, pipelineTask40)) + .thenReturn(List.of(pipelineTask30, pipelineTask35)); // Set up the DatastoreProducerConsumer retieval mocks. - Mockito - .when(datastoreProducerConsumerOperations.filesConsumedByTasks(List.of(30L, 40L), - scienceDatastoreFilenames)) - .thenReturn(Set.of( + when(datastoreProducerConsumerOperations.filesConsumedByTasks( + List.of(pipelineTask30, pipelineTask40), scienceDatastoreFilenames)).thenReturn(Set.of( "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-0.science.nc")); - Mockito - .when(datastoreProducerConsumerOperations.filesConsumedByTasks(List.of(30L, 35L), - collateralDatastoreFilenames)) - .thenReturn(Set.of( + when(datastoreProducerConsumerOperations.filesConsumedByTasks(List.of(pipelineTask30, + pipelineTask35), collateralDatastoreFilenames)).thenReturn(Set.of( "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-0.collateral.nc", "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-1.collateral.nc")); } diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java index c884344..312c6e2 100644 --- a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java @@ -13,7 +13,7 @@ import java.util.Map; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; diff --git a/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java b/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java index 18c9efa..b033965 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java @@ -31,6 +31,7 @@ import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.data.management.Acknowledgement.AcknowledgementEntry; import gov.nasa.ziggy.data.management.Manifest.ManifestEntry; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; @@ -78,7 +79,8 @@ public void testGenerateAcknowledgement() throws IOException { Manifest manifest = Manifest.generateManifest(testDataDir, 100L); manifest.setName("test-manifest.xml"); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); assertEquals("test-manifest-ack.xml", ack.getName()); assertEquals(17, ack.getFileCount()); assertEquals(100L, ack.getDatasetId()); @@ -103,7 +105,8 @@ public void testGenerateAcknowledgementFromSymlinks() throws IOException { Manifest manifest = Manifest.generateManifest(testDataDir, 100L); manifest.setName("test-manifest.xml"); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); assertEquals("test-manifest-ack.xml", ack.getName()); assertEquals(34, ack.getFileCount()); assertEquals(100L, ack.getDatasetId()); @@ -129,7 +132,8 @@ public void testFailedTransferStatus() throws IOException { Manifest manifest = Manifest.generateManifest(testDataDir, 100L); manifest.setName("test-manifest.xml"); manifest.getManifestEntries().get(0).setName("dummy-name.rpi"); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); assertEquals("test-manifest-ack.xml", ack.getName()); assertEquals(34, ack.getFileCount()); assertEquals(100L, ack.getDatasetId()); @@ -164,7 +168,8 @@ public void testFailedSizeValidation() throws IOException { String testFileName = manifest.getManifestEntries().get(0).getName(); long trueSize = manifest.getManifestEntries().get(0).getSize(); manifest.getManifestEntries().get(0).setSize(trueSize + 1); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); assertEquals("test-manifest-ack.xml", ack.getName()); assertEquals(34, ack.getFileCount()); assertEquals(100L, ack.getDatasetId()); @@ -199,7 +204,8 @@ public void testFailedChecksumValidation() throws IOException { String testFileName = manifest.getManifestEntries().get(0).getName(); String trueChecksum = manifest.getManifestEntries().get(0).getChecksum(); manifest.getManifestEntries().get(0).setChecksum(trueChecksum + "0"); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); assertEquals("test-manifest-ack.xml", ack.getName()); assertEquals(34, ack.getFileCount()); assertEquals(100L, ack.getDatasetId()); @@ -228,7 +234,8 @@ public void testXmlRoundTrip() Manifest manifest = Manifest.generateManifest(testDataDir, 100L); manifest.setName("test-manifest.xml"); - Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, 0); + Acknowledgement ack = Acknowledgement.of(manifest, testDataDir, + new PipelineTask(null, null, null)); ack.write(testDataDir); // There's no method for reading in an acknowledgement because diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java index c401aa6..33d8bf0 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java @@ -21,7 +21,7 @@ public static class PipelineInputsSample extends DatastoreDirectoryPipelineInput private double dvalue; /** - * Since the populateSubTaskInputs() method can do anything, we'll just have it set the + * Since the populateSubtaskInputs() method can do anything, we'll just have it set the * dvalue */ public void populateSubtaskInputs() { diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptExternalDefinitionTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptExternalDefinitionTest.java index e37cf48..8be3774 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptExternalDefinitionTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptExternalDefinitionTest.java @@ -51,7 +51,7 @@ public class DataReceiptExternalDefinitionTest { @Before public void setUp() { pipelineTask = Mockito.mock(PipelineTask.class); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(new UnitOfWork()); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(new UnitOfWork()); module = new TestDataReceiptPipelineModule(pipelineTask, RunMode.STANDARD); module = Mockito.spy(module); Mockito.doNothing().when(module).incrementProcessingStep(); diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptOperationsTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptOperationsTest.java index 627a513..4a3c38a 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptOperationsTest.java @@ -181,7 +181,7 @@ private ImportFiles setUpDataReceiptInstance(int successfulImports, int failedIm .merge(new PipelineInstanceNode(dataReceiptNode, dataReceiptModule)); instance1.addPipelineInstanceNode(instanceNode1); instance1 = new PipelineInstanceCrud().merge(instance1); - PipelineTask task1 = new PipelineTask(instance1, instanceNode1); + PipelineTask task1 = new PipelineTask(instance1, instanceNode1, null); task1 = new PipelineTaskCrud().merge(task1); instanceNode1.addPipelineTask(task1); instanceNode1 = new PipelineInstanceNodeCrud().merge(instanceNode1); diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java index 1db5ba6..637d89a 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java @@ -49,10 +49,10 @@ import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.database.ModelCrud; -import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceCrud; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.database.ModelCrud; +import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceCrud; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; @@ -195,7 +195,7 @@ public void testImportFromDataReceiptDir() throws IOException, InstantiationExce // Populate the importer files constructFilesForImport(dataImporterPath, true); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(singleUow); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, @@ -304,7 +304,7 @@ public void testImportFromDataSubdir() throws IOException, InstantiationExceptio // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(dataSubdirUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(dataSubdirUow); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, @@ -391,7 +391,7 @@ public void testImportWithErrors() throws IOException, InstantiationException, // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(singleUow); // Generate data and model importers that will throw IOExceptions at opportune moments Path dataReceiptExceptionPath = dataImporterPath.resolve(Paths.get("sector-0002", "mda", @@ -641,7 +641,7 @@ public void testCleanupFailOnNonEmptyDir() throws IOException, InstantiationExce // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(singleUow); Mockito.when(pipelineTask.getId()).thenReturn(101L); // Perform the import @@ -658,7 +658,7 @@ public void testImportOnEmptyDirectory() throws IOException { Files.delete(dataImporterPath); } Files.createDirectories(dataImporterPath); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(singleUow); constructDataReceiptDefinition(); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, @@ -675,7 +675,7 @@ public void testMissingManifest() throws InstantiationException, IllegalAccessEx constructFilesForImport(dataImporterPath, true); Files.delete(dataImporterPath.resolve("data-importer-manifest.xml")); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(singleUow); constructDataReceiptDefinition(); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, @@ -868,7 +868,7 @@ protected void persistProducerConsumerRecords(Collection successfulImports Collection failedImports) { for (Path file : successfulImports) { successfulImportsDataAccountability - .add(new DatastoreProducerConsumer(pipelineTask.getId(), file.toString())); + .add(new DatastoreProducerConsumer(pipelineTask, file)); } for (Path file : failedImports) { failedImportsDataAccountability.add(new FailedImport(pipelineTask, file)); diff --git a/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperationsTest.java b/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperationsTest.java index d7d7f68..9687b4e 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerOperationsTest.java @@ -143,7 +143,7 @@ public void testRetrieveFilesConsumedByTask() { new HashSet<>(Set.of(PATH_2.toString(), PATH_3.toString()))); // Retrieve the files that have the relevant pipeline task as consumer. - Set files = testOperations.filesConsumedByTask(31L); + Set files = testOperations.filesConsumedByTask(consumer1); assertEquals(2, files.size()); assertTrue(files.contains(PATH_1.toString())); assertTrue(files.contains(PATH_3.toString())); @@ -156,22 +156,31 @@ public void testRetrieveFilesConsumedByTasks() { datastoreProducerConsumerOperations.createOrUpdateProducer(pipelineTask, List.of(PATH_1, PATH_2, PATH_3)); - // Add comsumers. - Mockito.when(pipelineTask.getId()).thenReturn(100L); - datastoreProducerConsumerOperations.addConsumer(pipelineTask, Set.of(PATH_1.toString())); - Mockito.when(pipelineTask.getId()).thenReturn(110L); - datastoreProducerConsumerOperations.addConsumer(pipelineTask, Set.of(PATH_3.toString())); - Mockito.when(pipelineTask.getId()).thenReturn(120L); - datastoreProducerConsumerOperations.addConsumer(pipelineTask, Set.of(PATH_2.toString())); - - Set filenames = testOperations.filesConsumedByTasks(Set.of(100L, 105L), null); + // Add consumers. + PipelineTask pipelineTask100 = Mockito.mock(PipelineTask.class); + Mockito.when(pipelineTask100.getId()).thenReturn(100L); + datastoreProducerConsumerOperations.addConsumer(pipelineTask100, Set.of(PATH_1.toString())); + PipelineTask pipelineTask110 = Mockito.mock(PipelineTask.class); + Mockito.when(pipelineTask110.getId()).thenReturn(110L); + datastoreProducerConsumerOperations.addConsumer(pipelineTask110, Set.of(PATH_3.toString())); + PipelineTask pipelineTask120 = Mockito.mock(PipelineTask.class); + Mockito.when(pipelineTask120.getId()).thenReturn(120L); + datastoreProducerConsumerOperations.addConsumer(pipelineTask120, Set.of(PATH_2.toString())); + + PipelineTask pipelineTask105 = Mockito.mock(PipelineTask.class); + Mockito.when(pipelineTask105.getId()).thenReturn(105L); + + Set filenames = testOperations + .filesConsumedByTasks(Set.of(pipelineTask100, pipelineTask105), null); assertTrue(filenames.contains(PATH_1.toString())); assertEquals(1, filenames.size()); - filenames = testOperations.filesConsumedByTasks(Set.of(100L, 105L, 110L), null); + filenames = testOperations + .filesConsumedByTasks(Set.of(pipelineTask100, pipelineTask105, pipelineTask110), null); assertTrue(filenames.contains(PATH_1.toString())); assertTrue(filenames.contains(PATH_3.toString())); assertEquals(2, filenames.size()); - filenames = testOperations.filesConsumedByTasks(Set.of(100L, 105L, 110L), + filenames = testOperations.filesConsumedByTasks( + Set.of(pipelineTask100, pipelineTask105, pipelineTask110), Set.of(PATH_2.toString(), PATH_3.toString())); assertTrue(filenames.contains(PATH_3.toString())); assertEquals(1, filenames.size()); @@ -223,14 +232,15 @@ public Set producers(Set files) { () -> new DatastoreProducerConsumerCrud().retrieveProducers(files)); } - public Set filesConsumedByTask(long taskId) { - return performTransaction( - () -> new DatastoreProducerConsumerCrud().retrieveFilesConsumedByTask(taskId)); + public Set filesConsumedByTask(PipelineTask pipelineTask) { + return performTransaction(() -> new DatastoreProducerConsumerCrud() + .retrieveFilesConsumedByTask(pipelineTask)); } - public Set filesConsumedByTasks(Set taskIds, Collection filenames) { + public Set filesConsumedByTasks(Set pipelineTasks, + Collection filenames) { return performTransaction(() -> new DatastoreProducerConsumerCrud() - .retrieveFilesConsumedByTasks(taskIds, filenames)); + .retrieveFilesConsumedByTasks(pipelineTasks, filenames)); } } } diff --git a/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java b/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java index fc19e2b..7bf27dd 100644 --- a/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java +++ b/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java @@ -5,6 +5,7 @@ import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; import java.util.ArrayList; import java.util.Date; @@ -13,12 +14,16 @@ import org.junit.Rule; import org.junit.Test; +import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; +import gov.nasa.ziggy.uow.UnitOfWork; +import gov.nasa.ziggy.util.SystemProxy; public class TaskMetricsTest { @@ -29,14 +34,12 @@ public class TaskMetricsTest { @Rule public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); - @SuppressWarnings("unlikely-arg-type") @Test public void testHashCodeEquals() { TaskMetrics taskMetrics2 = taskMetrics(2); taskMetrics2.calculate(); assertTrue(taskMetrics2.equals(taskMetrics2)); assertFalse(taskMetrics2.equals(null)); - assertFalse(taskMetrics2.equals("a string")); TaskMetrics taskMetrics3 = taskMetrics(3); taskMetrics3.calculate(); @@ -81,18 +84,17 @@ public void testGetUnallocatedTime() { public void testGetTotalProcessingTimeMillis() { TaskMetrics taskMetrics = taskMetrics(3); taskMetrics.calculate(); - long totalProcessingTimeMillis = taskMetrics.getTotalProcessingTimeMillis(); - assertEquals(totalDuration, totalProcessingTimeMillis); + assertEquals(totalDuration, taskMetrics.getTotalProcessingTimeMillis()); } private TaskMetrics taskMetrics(int taskCount) { return new TaskMetrics(pipelineTasks(taskCount)); } - private List pipelineTasks(int taskCount) { + private List pipelineTasks(int taskCount) { // Each task starts one hour after the last. The task duration starts at one hour and each // subsequent task is one hour longer. - ArrayList pipelineTasks = new ArrayList<>(); + ArrayList pipelineTasks = new ArrayList<>(); long startTime = START_MILLIS; for (int i = 0; i < taskCount; i++) { long duration = (i + 1) * HOUR_MILLIS; @@ -104,15 +106,24 @@ private List pipelineTasks(int taskCount) { return pipelineTasks; } - private PipelineTask pipelineTask(String moduleName, Date start, Date end) { - PipelineTask pipelineTask = new PipelineTask(); - pipelineTask.setStartProcessingTime(start); - pipelineTask.setEndProcessingTime(end); - pipelineTask.setSummaryMetrics(summaryMetrics(moduleName)); - return new PipelineTaskOperations().merge(pipelineTask); + private PipelineTaskDisplayData pipelineTask(String moduleName, Date start, Date end) { + PipelineTask pipelineTask = Mockito + .spy(new PipelineTask(null, null, new UnitOfWork(moduleName))); + doReturn(42L).when(pipelineTask).getId(); + + PipelineTaskData pipelineTaskData = new PipelineTaskData(pipelineTask); + pipelineTaskData.setPipelineTaskMetrics(pipelineTaskMetrics(moduleName)); + PipelineTaskDisplayData pipelineTaskDisplayData = new PipelineTaskDisplayData( + pipelineTaskData); + SystemProxy.setUserTime(start.getTime()); + pipelineTaskDisplayData.getExecutionClock().start(); + SystemProxy.setUserTime(end.getTime()); + pipelineTaskDisplayData.getExecutionClock().stop(); + + return pipelineTaskDisplayData; } - private List summaryMetrics(String moduleName) { - return List.of(new PipelineTaskMetrics(moduleName, 42, Units.TIME)); + private List pipelineTaskMetrics(String moduleName) { + return List.of(new PipelineTaskMetric(moduleName, 42, Units.TIME)); } } diff --git a/src/test/java/gov/nasa/ziggy/metrics/report/MemdroneLogTest.java b/src/test/java/gov/nasa/ziggy/metrics/report/MemdroneLogTest.java index 23b98da..587f96c 100644 --- a/src/test/java/gov/nasa/ziggy/metrics/report/MemdroneLogTest.java +++ b/src/test/java/gov/nasa/ziggy/metrics/report/MemdroneLogTest.java @@ -22,7 +22,6 @@ /** * @author Todd Klaus */ -@Category(IntegrationTestCategory.class) public class MemdroneLogTest { private static final Logger log = LoggerFactory.getLogger(MemdroneLogTest.class); @@ -34,6 +33,7 @@ public class MemdroneLogTest { private static final int[] expectedSampleCounts = { 109003, 21253, 17881, 1, 42567, 25968, 15783, 209673, 85938, 10369, 1, 1, 13542 }; + @Category(IntegrationTestCategory.class) @Test public void testParse() throws Exception { File memdroneFile = new File(MEMDRONE_LOG_PATH); @@ -45,7 +45,7 @@ public void testParse() throws Exception { assertEquals("numKeys", expectedPids.length, processIds.size()); for (String pid : processIds) { - log.info("pid: " + pid + ", N=" + contents.get(pid).getN()); + log.info("pid={}, N={}", pid, contents.get(pid).getN()); } for (int i = 0; i < expectedPids.length; i++) { diff --git a/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java index ce3514c..23cbfa9 100644 --- a/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java @@ -9,6 +9,8 @@ import gov.nasa.ziggy.module.remote.nas.NasExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; /** @@ -42,7 +44,8 @@ public void testNewInstanceNullRemoteParameters() { .when( pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(new PipelineDefinitionNodeExecutionResources("dummy", "dummy")); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations, + new PipelineTaskDataOperations()); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -59,7 +62,8 @@ public void testNewInstanceRemoteDisabled() { .when( pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(executionResources); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations, + new PipelineTaskDataOperations()); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -78,9 +82,12 @@ public void testNewInstanceTooFewSubtasks() { .when( pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(executionResources); - Mockito.when(task.getTotalSubtaskCount()).thenReturn(100); - Mockito.when(task.getCompletedSubtaskCount()).thenReturn(99); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations); + PipelineTaskDataOperations pipelineTaskDataOperations = Mockito + .mock(PipelineTaskDataOperations.class); + SubtaskCounts subtaskCounts = new SubtaskCounts(100, 99, 0); + Mockito.when(pipelineTaskDataOperations.subtaskCounts(task)).thenReturn(subtaskCounts); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations, + pipelineTaskDataOperations); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -99,9 +106,12 @@ public void testNewInstanceRemote() { .when( pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(executionResources); - Mockito.when(task.getTotalSubtaskCount()).thenReturn(100); - Mockito.when(task.getCompletedSubtaskCount()).thenReturn(90); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations); + PipelineTaskDataOperations pipelineTaskDataOperations = Mockito + .mock(PipelineTaskDataOperations.class); + SubtaskCounts subtaskCounts = new SubtaskCounts(100, 90, 0); + Mockito.when(pipelineTaskDataOperations.subtaskCounts(task)).thenReturn(subtaskCounts); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, pipelineTaskOperations, + pipelineTaskDataOperations); assertTrue(executor instanceof NasExecutor); } } diff --git a/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java b/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java index c0a556e..bb5e6b8 100644 --- a/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java @@ -2,14 +2,24 @@ import static gov.nasa.ziggy.services.config.PropertyName.RESULTS_DIR; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; import java.io.IOException; import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; import java.util.List; +import java.util.Map; import org.apache.commons.configuration2.ex.ConfigurationException; -import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -17,20 +27,24 @@ import org.mockito.ArgumentMatchers; import org.mockito.Mockito; -import gov.nasa.ziggy.TestEventDetector; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; +import gov.nasa.ziggy.module.AlgorithmMonitor.Disposition; +import gov.nasa.ziggy.module.remote.PbsLogParser; +import gov.nasa.ziggy.module.remote.QueueCommandManager; +import gov.nasa.ziggy.module.remote.RemoteJobInformation; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; -import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceNodeOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.services.database.DatabaseService; +import gov.nasa.ziggy.services.messages.MonitorAlgorithmRequest; +import gov.nasa.ziggy.services.messages.TaskProcessingCompleteMessage; import gov.nasa.ziggy.supervisor.TaskRequestHandlerLifecycleManager; import gov.nasa.ziggy.supervisor.TaskRequestHandlerLifecycleManagerTest.InstrumentedTaskRequestHandlerLifecycleManager; @@ -38,19 +52,26 @@ * Unit tests for {@link AlgorithmMonitor}. * * @author PT + * @author Bill Wohler */ public class AlgorithmMonitorTest { - private AlgorithmMonitor monitor; - private PipelineTask pipelineTask; - private StateFile stateFile; - private JobMonitor jobMonitor; + private AlgorithmMonitorForTest monitor; + private PipelineTask pipelineTask100; + private PipelineTask pipelineTask101; + private QueueCommandManager queueCommandManager; + private PbsLogParser pbsLogParser; private PipelineExecutor pipelineExecutor; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; private PipelineInstanceNodeOperations pipelineInstanceNodeOperations; private AlertService alertService; - private PipelineDefinitionNodeExecutionResources resources = new PipelineDefinitionNodeExecutionResources( - "dummy", "dummy"); + private PipelineDefinitionNodeExecutionResources resources; + private List remoteJobsInformation; + private RemoteJobInformation job0Information; + private RemoteJobInformation job1Information; + private Path taskDirectoryTask100; + private Path taskDirectoryTask101; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); public TaskRequestHandlerLifecycleManager lifecycleManager = new InstrumentedTaskRequestHandlerLifecycleManager(); @@ -65,282 +86,307 @@ public class AlgorithmMonitorTest { @Before public void setUp() throws IOException, ConfigurationException { - DatabaseService.setInstance(Mockito.mock(DatabaseService.class)); - - Files.createDirectories(DirectoryProperties.stateFilesDir()); - jobMonitor = Mockito.mock(JobMonitor.class); - monitor = Mockito.spy(new AlgorithmMonitor(AlgorithmType.LOCAL)); - Mockito.when(monitor.jobMonitor()).thenReturn(jobMonitor); - Mockito.when(monitor.pollingIntervalMillis()).thenReturn(50L); - Mockito.doReturn(false).when(monitor).taskIsKilled(ArgumentMatchers.isA(long.class)); - pipelineTask = Mockito.spy(PipelineTask.class); - Mockito.doReturn(50L).when(pipelineTask).getPipelineInstanceId(); - Mockito.doReturn(50L).when(pipelineTask).getPipelineInstanceId(); - Mockito.doReturn(100L).when(pipelineTask).getId(); - Mockito.doReturn("dummy").when(pipelineTask).getModuleName(); - pipelineExecutor = Mockito.spy(PipelineExecutor.class); - pipelineTaskOperations = Mockito.mock(PipelineTaskOperations.class); - Mockito.when(pipelineTaskOperations.pipelineTask(100L)).thenReturn(pipelineTask); - Mockito.when(pipelineTaskOperations.merge(ArgumentMatchers.isA(PipelineTask.class))) - .thenReturn(pipelineTask); - pipelineInstanceNodeOperations = Mockito.mock(PipelineInstanceNodeOperations.class); - Mockito.doReturn(pipelineTaskOperations).when(pipelineExecutor).pipelineTaskOperations(); - Mockito.doReturn(pipelineInstanceNodeOperations) - .when(pipelineExecutor) + monitor = spy(new AlgorithmMonitorForTest()); + queueCommandManager = mock(QueueCommandManager.class); + pbsLogParser = mock(PbsLogParser.class); + resources = new PipelineDefinitionNodeExecutionResources("dummy", "dummy"); + remoteJobsInformation = new ArrayList<>(); + + doReturn(queueCommandManager).when(monitor).queueCommandManager(); + doReturn(0L).when(monitor).finishedJobsPollingIntervalMillis(); + doReturn(0L).when(monitor).localPollIntervalMillis(); + doReturn(0L).when(monitor).remotePollIntervalMillis(); + doReturn(false).when(monitor).taskIsKilled(ArgumentMatchers.isA(PipelineTask.class)); + pipelineTask100 = mock(PipelineTask.class); + doReturn(50L).when(pipelineTask100).getPipelineInstanceId(); + doReturn(50L).when(pipelineTask100).getPipelineInstanceId(); + doReturn(100L).when(pipelineTask100).getId(); + doReturn("dummy").when(pipelineTask100).getModuleName(); + pipelineTask101 = mock(PipelineTask.class); + doReturn(101L).when(pipelineTask101).getId(); + pipelineExecutor = mock(PipelineExecutor.class); + pipelineTaskOperations = mock(PipelineTaskOperations.class); + when(pipelineTaskOperations.pipelineTask(100L)).thenReturn(pipelineTask100); + when(pipelineTaskOperations.merge(ArgumentMatchers.isA(PipelineTask.class))) + .thenReturn(pipelineTask100); + pipelineTaskDataOperations = mock(PipelineTaskDataOperations.class); + pipelineInstanceNodeOperations = mock(PipelineInstanceNodeOperations.class); + doReturn(pipelineTaskOperations).when(pipelineExecutor).pipelineTaskOperations(); + doReturn(pipelineInstanceNodeOperations).when(pipelineExecutor) .pipelineInstanceNodeOperations(); - Mockito.when(monitor.pipelineTaskOperations()).thenReturn(pipelineTaskOperations); + doReturn(pipelineTaskOperations).when(monitor).pipelineTaskOperations(); + doReturn(pipelineTaskDataOperations).when(monitor).pipelineTaskDataOperations(); Mockito.doNothing() .when(pipelineExecutor) - .removeTaskFromKilledTaskList(ArgumentMatchers.isA(long.class)); - Mockito.when(pipelineExecutor.taskRequestEnabled()).thenReturn(false); - Mockito - .when( - pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) + .removeTaskFromKilledTaskList(ArgumentMatchers.isA(PipelineTask.class)); + when(pipelineExecutor.taskRequestEnabled()).thenReturn(false); + when(pipelineTaskOperations.executionResources(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(resources); - Mockito.when(monitor.pipelineExecutor()).thenReturn(pipelineExecutor); - Mockito.when(monitor.pipelineTaskOperations()).thenReturn(pipelineTaskOperations); - Mockito.when(pipelineTaskOperations.pipelineTask(ArgumentMatchers.anyLong())) - .thenReturn(pipelineTask); - Mockito.when(pipelineTaskOperations.merge(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn(pipelineTask); - Mockito - .when(pipelineTaskOperations - .prepareTaskForAutoResubmit(ArgumentMatchers.any(PipelineTask.class))) - .thenReturn(pipelineTask); - alertService = Mockito.mock(AlertService.class); - Mockito.when(monitor.alertService()).thenReturn(alertService); - stateFile = StateFile.generateStateFile(pipelineTask, null, 100); - stateFile.persist(); - monitor.startMonitoring(stateFile); + when(monitor.pipelineExecutor()).thenReturn(pipelineExecutor); + when(monitor.pipelineTaskOperations()).thenReturn(pipelineTaskOperations); + when(pipelineTaskOperations.pipelineTask(ArgumentMatchers.anyLong())) + .thenReturn(pipelineTask100); + alertService = mock(AlertService.class); + when(monitor.alertService()).thenReturn(alertService); + when(monitor.pbsLogParser()).thenReturn(pbsLogParser); + taskDirectoryTask100 = Files + .createDirectories(DirectoryProperties.taskDataDir().resolve("50-100-dummy")); + Files.createDirectories(taskDirectoryTask100.resolve(SubtaskUtils.subtaskDirName(0))); + job0Information = new RemoteJobInformation( + DirectoryProperties.pbsLogDir().toAbsolutePath().resolve("pbsLogFile1").toString(), + "50-100-dummy.0"); + job0Information.setJobId(1L); + job1Information = new RemoteJobInformation( + DirectoryProperties.pbsLogDir().toAbsolutePath().resolve("pbsLogFile2").toString(), + "50-100-dummy.1"); + job1Information.setJobId(2L); + remoteJobsInformation.add(job0Information); + remoteJobsInformation.add(job1Information); + taskDirectoryTask101 = Files + .createDirectories(DirectoryProperties.taskDataDir().resolve("50-101-dummy")); + Files.createDirectories(taskDirectoryTask101.resolve(SubtaskUtils.subtaskDirName(0))); + Files.createDirectories(DirectoryProperties.pbsLogDir()); } - @After - public void tearDown() throws IOException { - DatabaseService.reset(); - TestEventDetector.detectTestEvent(500L, () -> (lifecycleManager.taskRequestSize() == 0)); + @Test + public void testAddLocalTaskToMonitor() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + assertTrue(monitor.getJobsInformationByTask().isEmpty()); + Map taskMonitorByTaskId = monitor.getTaskMonitorByTask(); + assertNotNull(taskMonitorByTaskId.get(pipelineTask100)); + TaskMonitor taskMonitor = taskMonitorByTaskId.get(pipelineTask100); + assertTrue(taskMonitor.getSubtaskDirectories() + .contains(taskDirectoryTask101.resolve(SubtaskUtils.subtaskDirName(0)))); + assertEquals(1, taskMonitor.getSubtaskDirectories().size()); + assertEquals(taskDirectoryTask101, taskMonitor.getTaskDir()); + assertEquals(1, taskMonitorByTaskId.size()); + assertTrue(monitor.getJobsInformationByTask().isEmpty()); } - /** - * Executes the {@link AlgorithmMonitor#run()} method a fixed number of times. This is necessary - * because in some cases a later pass through run() is needed to respond to an action taken in - * an earlier pass. - * - * @param iterationCount number of executions of run. - */ - private void iterateAlgorithmMonitorRunMethod(int iterationCount) { - for (int i = 0; i < iterationCount; i++) { - monitor.run(); - } + @Test + public void testAddRemoteTaskToMonitor() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + Map taskMonitorByTaskId = monitor.getTaskMonitorByTask(); + assertNotNull(taskMonitorByTaskId.get(pipelineTask100)); + TaskMonitor taskMonitor = taskMonitorByTaskId.get(pipelineTask100); + assertTrue(taskMonitor.getSubtaskDirectories() + .contains(taskDirectoryTask100.resolve(SubtaskUtils.subtaskDirName(0)))); + assertEquals(1, taskMonitor.getSubtaskDirectories().size()); + assertEquals(taskDirectoryTask100, taskMonitor.getTaskDir()); + assertEquals(1, taskMonitorByTaskId.size()); + assertNotNull(monitor.getJobsInformationByTask().get(pipelineTask100)); + assertTrue( + monitor.getJobsInformationByTask().get(pipelineTask100).contains(job0Information)); + assertTrue( + monitor.getJobsInformationByTask().get(pipelineTask100).contains(job1Information)); + assertEquals(1, monitor.getJobsInformationByTask().size()); } - // Tests basic "does the state file get into the Map?" functionality. @Test - public void testStateFileSubmission() { - StateFile storedStateFile = monitor.getStateFile(stateFile); - assertEquals(50L, storedStateFile.getPipelineInstanceId()); - assertEquals(100L, storedStateFile.getPipelineTaskId()); - assertEquals("dummy", storedStateFile.getModuleName()); - assertEquals(100, storedStateFile.getNumTotal()); - assertEquals(0, storedStateFile.getNumComplete()); - assertEquals(0, storedStateFile.getNumFailed()); - assertEquals(StateFile.State.QUEUED, storedStateFile.getState()); + public void testJobIdsByTaskId() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + Map> jobIdsByTaskId = monitor + .jobIdsByTaskId(List.of(pipelineTask101)); + assertNotNull(jobIdsByTaskId); + assertTrue(jobIdsByTaskId.isEmpty()); + jobIdsByTaskId = monitor.jobIdsByTaskId(List.of(pipelineTask100)); + assertFalse(jobIdsByTaskId.isEmpty()); + assertNotNull(jobIdsByTaskId.get(pipelineTask100)); + assertTrue(jobIdsByTaskId.get(pipelineTask100).contains(1L)); + assertTrue(jobIdsByTaskId.get(pipelineTask100).contains(2L)); + assertEquals(2, jobIdsByTaskId.get(pipelineTask100).size()); + assertEquals(1, jobIdsByTaskId.size()); } - // Tests that an update of the state file on disk gets reflected in the stored - // state file. @Test - public void testStateFileUpdate() - throws ConfigurationException, IOException, InterruptedException { - stateFile.setState(StateFile.State.PROCESSING); - stateFile.setNumComplete(10); - stateFile.setNumFailed(5); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(1); - StateFile storedStateFile = monitor.getStateFile(stateFile); - assertEquals(50L, storedStateFile.getPipelineInstanceId()); - assertEquals(100L, storedStateFile.getPipelineTaskId()); - assertEquals("dummy", storedStateFile.getModuleName()); - assertEquals(100, storedStateFile.getNumTotal()); - assertEquals(10, storedStateFile.getNumComplete()); - assertEquals(5, storedStateFile.getNumFailed()); - assertEquals(StateFile.State.PROCESSING, storedStateFile.getState()); - Mockito.verify(pipelineTaskOperations).updateProcessingStep(100L, ProcessingStep.EXECUTING); + public void testJobIdsByTaskIdOneJobFinished() throws IOException { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + Files.createFile(DirectoryProperties.pbsLogDir().resolve(job0Information.getLogFile())); + Map> jobIdsByTaskId = monitor + .jobIdsByTaskId(List.of(pipelineTask100)); + assertFalse(jobIdsByTaskId.isEmpty()); + assertNotNull(jobIdsByTaskId.get(pipelineTask100)); + assertTrue(jobIdsByTaskId.get(pipelineTask100).contains(2L)); + assertEquals(1, jobIdsByTaskId.get(pipelineTask100).size()); + assertEquals(1, jobIdsByTaskId.size()); } - // Test a failed execution, that is to say one in which: - // (a) The JobMonitor detects that the task failed, and - // (b) the number of failed subtasks exceeds the number allowed for a - // declaration of victory. @Test - public void testExecutionFailed() - throws ConfigurationException, IOException, InterruptedException { - resources.setMaxFailedSubtaskCount(4); - stateFile.setState(StateFile.State.PROCESSING); - stateFile.setNumComplete(90); - stateFile.setNumFailed(5); - stateFile.persist(); - Mockito.when(jobMonitor.isFinished(ArgumentMatchers.any(StateFile.class))).thenReturn(true); - iterateAlgorithmMonitorRunMethod(3); - - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); - - // The state file on disk is moved to state COMPLETE. - StateFile updatedStateFile = stateFile.newStateFileFromDiskFile(); - assertEquals(StateFile.State.COMPLETE, updatedStateFile.getState()); - - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); - - // The pipeline task state was set to ERROR. - Mockito.verify(pipelineTaskOperations).taskErrored(pipelineTask); - - // There are no task requests in the queue. - assertEquals(0, lifecycleManager.taskRequestSize()); + public void testEndMonitoringNoSuchTask() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitor.endTaskMonitoring(pipelineTask101); + assertNull(monitor.getDisposition()); + Mockito.verify(pipelineTaskDataOperations, Mockito.times(0)) + .updateProcessingStep(pipelineTask101, ProcessingStep.WAITING_TO_STORE); + Mockito.verify(pipelineTaskDataOperations, Mockito.times(0)) + .updateJobs(pipelineTask100, true); + Mockito.verify(queueCommandManager, Mockito.times(0)) + .deleteJobsByJobId(ArgumentMatchers.anyList()); + assertNotNull(monitor.getJobsInformationByTask().get(pipelineTask100)); + assertNotNull(monitor.getTaskMonitorByTask().get(pipelineTask100)); } - // Tests an execution that is complete, but has too many errors to be persisted. @Test - public void testExecutionCompleteTooManyErrors() - throws ConfigurationException, IOException, InterruptedException { - - resources.setMaxFailedSubtaskCount(4); - stateFile.setState(StateFile.State.COMPLETE); - stateFile.setNumComplete(95); - stateFile.setNumFailed(5); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(2); - - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); - - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); - - // The pipeline task state was set to ERROR. - Mockito.verify(pipelineTaskOperations).taskErrored(pipelineTask); - - // There are no task requests in the queue. - assertEquals(0, lifecycleManager.taskRequestSize()); + public void testEndMonitoringLocalTaskSuccessful() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask101)) + .thenReturn(new SubtaskCounts(1, 1, 0)); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask101); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.PERSIST, monitor.getDisposition()); + Mockito.verify(pipelineExecutor).persistTaskResults(pipelineTask101); + assertNotNull(monitor.getTaskMonitorByTask().get(pipelineTask100)); + assertNull(monitor.getTaskMonitorByTask().get(pipelineTask101)); + assertNotNull(monitor.getJobsInformationByTask().get(pipelineTask100)); + Mockito.verify(queueCommandManager, Mockito.times(0)) + .deleteJobsByJobId(ArgumentMatchers.anyList()); + Mockito.verify(pipelineTaskDataOperations, Mockito.times(0)) + .updateJobs(pipelineTask100, true); } - // Tests an execution that is complete, but has too many unprocessed subtasks to - // declare victory. @Test - public void testExecutionCompleteTooManyMissed() - throws ConfigurationException, IOException, InterruptedException { - - resources.setMaxFailedSubtaskCount(4); - stateFile.setState(StateFile.State.COMPLETE); - stateFile.setNumComplete(95); - stateFile.setNumFailed(0); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(2); - - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); - - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); - - // The pipeline task state was set to ERROR. - Mockito.verify(pipelineTaskOperations).taskErrored(pipelineTask); - - // There are no task requests in the queue. - assertEquals(0, lifecycleManager.taskRequestSize()); + public void testEndMonitoringRemoteTaskSuccessful() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask100)) + .thenReturn(new SubtaskCounts(1, 1, 0)); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask100); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.PERSIST, monitor.getDisposition()); + Mockito.verify(pipelineExecutor).persistTaskResults(pipelineTask100); + assertNull(monitor.getTaskMonitorByTask().get(pipelineTask100)); + assertNotNull(monitor.getTaskMonitorByTask().get(pipelineTask101)); + assertNull(monitor.getJobsInformationByTask().get(pipelineTask100)); + Mockito.verify(queueCommandManager).deleteJobsByJobId(ArgumentMatchers.anyList()); + Mockito.verify(pipelineTaskDataOperations).updateJobs(pipelineTask100, true); } - // Tests that when execution is COMPLETE and the number of failed subtasks is - // small enough, a request to persist the task results is issued. @Test - public void testExecutionComplete() - throws ConfigurationException, IOException, InterruptedException { - - resources.setMaxFailedSubtaskCount(6); - stateFile.setState(StateFile.State.COMPLETE); - stateFile.setNumComplete(95); - stateFile.setNumFailed(5); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(2); - - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); - - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); - - // The PipelineExecutor should have been asked to submit the task for persisting. - Mockito.verify(pipelineExecutor).persistTaskResults(pipelineTask); + public void testDispositionHaltedTask() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask100)) + .thenReturn(new SubtaskCounts(1, 1, 0)); + doReturn(true).when(monitor).taskIsKilled(pipelineTask100); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask100); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.FAIL, monitor.getDisposition()); } - // Test automatic resubmission of a task that's almost but not quite finished - // according to the number of failed subtasks. @Test - public void testAutoResubmit() - throws ConfigurationException, IOException, InterruptedException { + public void testDispositionTaskWithAcceptableErrors() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask100)) + .thenReturn(new SubtaskCounts(1, 0, 1)); + doReturn(false).when(monitor).taskIsKilled(pipelineTask100); + resources.setMaxFailedSubtaskCount(1); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask100); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.PERSIST, monitor.getDisposition()); + } - int taskCount = lifecycleManager.taskRequestSize(); - assertEquals(0, taskCount); - resources.setMaxFailedSubtaskCount(4); - resources.setMaxAutoResubmits(3); - Mockito.when(pipelineTask.getAutoResubmitCount()).thenReturn(1); - Mockito.when(pipelineTask.isError()).thenReturn(true); + @Test + public void testResubmitDisposition() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask100)) + .thenReturn(new SubtaskCounts(1, 0, 1)); + when(pipelineTaskDataOperations.autoResubmitCount(pipelineTask100)).thenReturn(0); + doReturn(false).when(monitor).taskIsKilled(pipelineTask100); + resources.setMaxAutoResubmits(1); Mockito.doNothing() .when(pipelineExecutor) - .restartFailedTasks(ArgumentMatchers.anyCollection(), ArgumentMatchers.anyBoolean(), - ArgumentMatchers.isA(RunMode.class)); - stateFile.setState(StateFile.State.COMPLETE); - stateFile.setNumComplete(95); - stateFile.setNumFailed(5); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(3); - - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); - - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); - - // The task should have its auto-resubmit count incremented - Mockito.verify(pipelineTaskOperations).prepareTaskForAutoResubmit(pipelineTask); - - // The pipeline executor method to restart tasks was called - Mockito.verify(pipelineExecutor) - .restartFailedTasks(List.of(pipelineTask), false, RunMode.RESUBMIT); + .restartFailedTasks(ArgumentMatchers.anyList(), ArgumentMatchers.anyBoolean(), + ArgumentMatchers.any()); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask100); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.RESUBMIT, monitor.getDisposition()); } - // Test the case where automatic submission would be called except that the - // task is out of automatic resubmits. @Test - public void testOutOfAutoResubmits() - throws ConfigurationException, IOException, InterruptedException { + public void testFailedDisposition() { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitorAlgorithmRequest = new MonitorAlgorithmRequest(pipelineTask101, + taskDirectoryTask101); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask100)) + .thenReturn(new SubtaskCounts(1, 0, 1)); + doReturn(false).when(monitor).taskIsKilled(pipelineTask100); + TaskProcessingCompleteMessage taskProcessingCompleteMessage = new TaskProcessingCompleteMessage( + pipelineTask100); + monitor.setTaskProcessingCompleteMessage(taskProcessingCompleteMessage); + assertEquals(Disposition.FAIL, monitor.getDisposition()); + } - resources.setMaxFailedSubtaskCount(4); - resources.setMaxAutoResubmits(3); - Mockito.when(pipelineTask.getAutoResubmitCount()).thenReturn(3); - Mockito.when(pipelineTask.isError()).thenReturn(true); - stateFile.setState(StateFile.State.COMPLETE); - stateFile.setNumComplete(95); - stateFile.setNumFailed(5); - stateFile.persist(); - iterateAlgorithmMonitorRunMethod(3); + @Test + public void testFinishedJobs() throws IOException { + MonitorAlgorithmRequest monitorAlgorithmRequest = new MonitorAlgorithmRequest( + pipelineTask100, taskDirectoryTask100, remoteJobsInformation); + monitor.setMonitorAlgorithmRequest(monitorAlgorithmRequest); + monitor.run(); + assertNull(monitor.allJobsFinishedMessage()); + Files.createFile(Paths.get(job0Information.getLogFile())); + monitor.run(); + assertNull(monitor.allJobsFinishedMessage()); + Files.createFile(Paths.get(job1Information.getLogFile())); + monitor.run(); + assertNotNull(monitor.allJobsFinishedMessage()); + // TODO Resurrect if AllJobsFinishedMessage.pipelineTask becomes non-transient + // assertEquals(pipelineTask100, monitor.allJobsFinishedMessage().getPipelineTask()); + } - // No state file remains in the monitoring system. - assertNull(monitor.getStateFile(stateFile)); + private static class AlgorithmMonitorForTest extends AlgorithmMonitor { - // The task should be advanced to Ac state - Mockito.verify(pipelineTaskOperations) - .updateProcessingStep(100L, ProcessingStep.WAITING_TO_STORE); + // Required for Mockito. + @SuppressWarnings("unused") + public AlgorithmMonitorForTest() { + } - // The pipeline task state was set to ERROR, then to SUBMITTED - Mockito.verify(pipelineTaskOperations).taskErrored(pipelineTask); + private void setMonitorAlgorithmRequest(MonitorAlgorithmRequest request) { + addToMonitor(request); + } - // There are no task requests in the queue. - assertEquals(0, lifecycleManager.taskRequestSize()); + private void setTaskProcessingCompleteMessage(TaskProcessingCompleteMessage message) { + endTaskMonitoring(message.getPipelineTask()); + } } } diff --git a/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java b/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java index a433104..f1cc6a2 100644 --- a/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java +++ b/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java @@ -2,12 +2,10 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; -import static org.mockito.Mockito.doNothing; import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; import java.io.File; import java.io.IOException; @@ -28,10 +26,10 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.remote.TimestampFile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.util.BuildInfo; +import gov.nasa.ziggy.util.BuildInfo.BuildType; /** * Unit tests for the {@link ComputeNodeMaster} class. @@ -45,23 +43,36 @@ public class ComputeNodeMasterTest { public ZiggyPropertyRule resultsDirRule = new ZiggyPropertyRule(PropertyName.RESULTS_DIR, directoryRule); + public ZiggyPropertyRule homeDirRule = new ZiggyPropertyRule(PropertyName.ZIGGY_HOME_DIR, + directoryRule); + + public ZiggyPropertyRule pipelineHomeDirRule = new ZiggyPropertyRule( + PropertyName.PIPELINE_HOME_DIR, directoryRule); + + @Rule + public RuleChain ruleChain = RuleChain.outerRule(directoryRule) + .around(resultsDirRule) + .around(homeDirRule) + .around(pipelineHomeDirRule); + + private final String MODULE_NAME = "dummy"; + @Rule - public RuleChain ruleChain = RuleChain.outerRule(directoryRule).around(resultsDirRule); + public ZiggyPropertyRule executableNameRule = new ZiggyPropertyRule( + PropertyName.ZIGGY_ALGORITHM_NAME, MODULE_NAME); private final int SUBTASK_COUNT = 5; private final int CORES_PER_NODE = 3; private final long INSTANCE_ID = 10; private final long TASK_ID = 20; - private final String MODULE_NAME = "dummy"; private final String TASK_DIR_NAME = Long.toString(INSTANCE_ID) + "-" + Long.toString(TASK_ID) + "-" + MODULE_NAME; + private final long WALL_TIME_REQUEST = 1800L; - private PipelineTask pipelineTask; private TaskConfiguration inputsHandler; private SubtaskServer subtaskServer; private ExecutorService subtaskMasterThreadPool; private ComputeNodeMaster computeNodeMaster; - private StateFile stateFile; private Path taskDir; private List subtaskDirFiles; @@ -79,19 +90,6 @@ public void setUp() throws Exception { Files.createDirectories(subtaskDir); } - // Create and populate the state file directory. - Files.createDirectories(DirectoryProperties.stateFilesDir()); - pipelineTask = mock(PipelineTask.class); - when(pipelineTask.getModuleName()).thenReturn(MODULE_NAME); - when(pipelineTask.getId()).thenReturn(TASK_ID); - when(pipelineTask.getPipelineInstanceId()).thenReturn(INSTANCE_ID); - stateFile = StateFile.generateStateFile(pipelineTask, null, SUBTASK_COUNT); - stateFile.setActiveCoresPerNode(CORES_PER_NODE); - stateFile.persist(); - - // Create the state file lock file - stateFile.lockFile().createNewFile(); - // Create the algorithm log directory and the TaskLog instance Files.createDirectories(DirectoryProperties.algorithmLogsDir()); @@ -100,38 +98,31 @@ public void setUp() throws Exception { subtaskServer = mock(SubtaskServer.class); subtaskMasterThreadPool = mock(ExecutorService.class); + // Create the wall time and active cores files. + AlgorithmExecutor.writeActiveCoresFile(taskDir, Integer.toString(CORES_PER_NODE)); + AlgorithmExecutor.writeWallTimeFile(taskDir, Long.toString(WALL_TIME_REQUEST)); + // Create the ComputeNodeMaster. To be precise, create an instance of the // class that is a Mockito spy. computeNodeMaster = Mockito.spy(new ComputeNodeMaster(taskDir.toString())); doReturn(inputsHandler).when(computeNodeMaster).getTaskConfiguration(); doReturn(subtaskServer).when(computeNodeMaster).subtaskServer(); doReturn(subtaskMasterThreadPool).when(computeNodeMaster).subtaskMasterThreadPool(); - doReturn(true).when(computeNodeMaster) - .getWriteLockWithoutBlocking(ArgumentMatchers.any(File.class)); - doNothing().when(computeNodeMaster).releaseWriteLock(ArgumentMatchers.any(File.class)); + + // Create the version information properties file. + new BuildInfo(BuildType.ZIGGY).writeBuildFile(); } /** - * Tests an {@link ComputeNodeMaster#initialize()} call that has to update the state file. + * Tests an {@link ComputeNodeMaster#initialize()} call. */ @Test - public void testInitializeAndUpdateState() + public void testInitialize() throws ConfigurationException, IllegalStateException, IOException, InterruptedException { - // The initial state of the state file should be QUEUED. - assertEquals(StateFile.State.QUEUED, stateFile.newStateFileFromDiskFile().getState()); - // Initialize the instance. computeNodeMaster.initialize(); - // The state file on disk should now be PROCESSING - assertEquals(StateFile.State.PROCESSING, stateFile.newStateFileFromDiskFile().getState()); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(0, computeNodeMaster.getStateFileNumComplete()); - assertEquals(0, computeNodeMaster.getStateFileNumFailed()); - // The SubtaskServer should have started verify(subtaskServer).start(); @@ -139,14 +130,11 @@ public void testInitializeAndUpdateState() assertEquals(3, computeNodeMaster.subtaskMastersCount()); verify(subtaskMasterThreadPool, times(3)).submit(ArgumentMatchers.any(SubtaskMaster.class), ArgumentMatchers.any(ThreadFactory.class)); - assertEquals(0, computeNodeMaster.getSemaphorePermits()); // There should be timestamp files in the task directory. - assertTrue(TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.ARRIVE_PFE) > 0); - assertTrue( - TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.PBS_JOB_START) > 0); - assertEquals(-1L, - TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.QUEUED_PBS)); + assertTrue(TimestampFile.timestamp(taskDir.toFile(), + TimestampFile.Event.ARRIVE_COMPUTE_NODES) > 0); + assertTrue(TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.START) > 0); } /** @@ -156,20 +144,9 @@ public void testInitializeAndUpdateState() public void testInitializeWithoutUpdatingState() throws IOException, ConfigurationException, IllegalStateException, InterruptedException { - doReturn(false).when(computeNodeMaster) - .getWriteLockWithoutBlocking(ArgumentMatchers.any(File.class)); - // Initialize the instance. computeNodeMaster.initialize(); - // The state file on disk should automatically increment to PROCESSING - assertEquals(StateFile.State.PROCESSING, stateFile.newStateFileFromDiskFile().getState()); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(0, computeNodeMaster.getStateFileNumComplete()); - assertEquals(0, computeNodeMaster.getStateFileNumFailed()); - // The SubtaskServer should have started verify(subtaskServer).start(); @@ -177,214 +154,11 @@ public void testInitializeWithoutUpdatingState() assertEquals(3, computeNodeMaster.subtaskMastersCount()); verify(subtaskMasterThreadPool, times(3)).submit(ArgumentMatchers.any(SubtaskMaster.class), ArgumentMatchers.any(ThreadFactory.class)); - assertEquals(0, computeNodeMaster.getSemaphorePermits()); // There should be timestamp files in the task directory. - assertTrue(TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.ARRIVE_PFE) > 0); - assertTrue( - TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.PBS_JOB_START) > 0); - assertEquals(-1L, - TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.QUEUED_PBS)); - } - - /** - * Tests an {@link ComputeNodeMaster#initialize()} call that runs after all subtasks have - * already been completed. - */ - @Test - public void testInitializeAllSubtasksDone() - throws IOException, ConfigurationException, IllegalStateException, InterruptedException { - - // Set all subtasks to be either FAILED or COMPLETE. - for (int subtask = 0; subtask < SUBTASK_COUNT; subtask++) { - File subtaskDirFile = taskDir.resolve("st-" + subtask).toFile(); - AlgorithmStateFiles.SubtaskState subtaskState = subtask == 0 - ? AlgorithmStateFiles.SubtaskState.FAILED - : AlgorithmStateFiles.SubtaskState.COMPLETE; - new AlgorithmStateFiles(subtaskDirFile).updateCurrentState(subtaskState); - } - - // Initialize the instance. - computeNodeMaster.initialize(); - - // The state file on disk should be COMPLETE - assertEquals(StateFile.State.COMPLETE, stateFile.newStateFileFromDiskFile().getState()); - - // The SubtaskServer should not have started - verify(subtaskServer, times(0)).start(); - - // There should be no SubtaskMaster instances. - assertEquals(0, computeNodeMaster.subtaskMastersCount()); - verify(subtaskMasterThreadPool, times(0)).submit(ArgumentMatchers.any(SubtaskMaster.class), - ArgumentMatchers.any(ThreadFactory.class)); - assertEquals(-1, computeNodeMaster.getSemaphorePermits()); - } - - /** - * Tests the performance of the monitoring process when there are subtasks that remain that - * require processing. - */ - @Test - public void testMonitoringWhenSubtasksRemain() - throws ConfigurationException, IllegalStateException, IOException, InterruptedException { - - when(subtaskServer.isListenerRunning()).thenReturn(true); - - // Initialize the instance. - computeNodeMaster.initialize(); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(0, computeNodeMaster.getStateFileNumComplete()); - assertEquals(0, computeNodeMaster.getStateFileNumFailed()); - - // The countdown latch should still be waiting. - assertEquals(1, computeNodeMaster.getCountDownLatchCount()); - - // All of the semaphore permits should still be in use. - assertEquals(0, computeNodeMaster.getSemaphorePermits()); - - // Mark a subtask as complete, another as processing, another as failed. - new AlgorithmStateFiles(taskDir.resolve("st-0").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(taskDir.resolve("st-1").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - new AlgorithmStateFiles(taskDir.resolve("st-2").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.PROCESSING); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(1, computeNodeMaster.getStateFileNumComplete()); - assertEquals(1, computeNodeMaster.getStateFileNumFailed()); - - // The countdown latch should still be waiting. - assertEquals(1, computeNodeMaster.getCountDownLatchCount()); - - // All of the semaphore permits should still be in use. - assertEquals(0, computeNodeMaster.getSemaphorePermits()); - } - - /** - * Tests the performance of the monitoring process when the {@link SubtaskServer} listener - * thread fails. - */ - @Test - public void testMonitoringWhenServerFails() - throws ConfigurationException, IllegalStateException, IOException, InterruptedException { - - // Initialize the instance. - computeNodeMaster.initialize(); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(0, computeNodeMaster.getStateFileNumComplete()); - assertEquals(0, computeNodeMaster.getStateFileNumFailed()); - - // The countdown latch should no longer be waiting. - assertEquals(0, computeNodeMaster.getCountDownLatchCount()); - - // All of the semaphore permits should still be in use. - assertEquals(0, computeNodeMaster.getSemaphorePermits()); - - // Mark a subtask as complete, another as processing, another as failed. - new AlgorithmStateFiles(taskDir.resolve("st-0").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(taskDir.resolve("st-1").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - new AlgorithmStateFiles(taskDir.resolve("st-2").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.PROCESSING); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The state counts should not reflect the changes because the - // monitoring system doesn't update the ComputeNodeMaster state - // when the countdown latch has been released. - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(0, computeNodeMaster.getStateFileNumComplete()); - assertEquals(0, computeNodeMaster.getStateFileNumFailed()); - - // The countdown latch should no longer be waiting. - assertEquals(0, computeNodeMaster.getCountDownLatchCount()); - - // All of the semaphore permits should still be in use. - assertEquals(0, computeNodeMaster.getSemaphorePermits()); - } - - /** - * Tests the performance of the monitoring process when all the subtasks are completed. - */ - @Test - public void testMonitoringCompletedTask() - throws ConfigurationException, IllegalStateException, IOException, InterruptedException { - - when(subtaskServer.isListenerRunning()).thenReturn(true); - - // Initialize the instance. - computeNodeMaster.initialize(); - - // Mark subtasks as complete or failed. - new AlgorithmStateFiles(taskDir.resolve("st-0").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(taskDir.resolve("st-1").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - new AlgorithmStateFiles(taskDir.resolve("st-2").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(taskDir.resolve("st-3").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - new AlgorithmStateFiles(taskDir.resolve("st-4").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The countdown latch should no longer be waiting. - assertEquals(0, computeNodeMaster.getCountDownLatchCount()); - - // All of the semaphore permits should still be in use. - assertEquals(0, computeNodeMaster.getSemaphorePermits()); - - // The state of the state file should still be processing. - assertEquals(StateFile.State.PROCESSING, stateFile.newStateFileFromDiskFile().getState()); - - // The state file in the ComputeNodeMaster should have correct counts - assertEquals(5, computeNodeMaster.getStateFileNumTotal()); - assertEquals(3, computeNodeMaster.getStateFileNumComplete()); - assertEquals(2, computeNodeMaster.getStateFileNumFailed()); - } - - /** - * Tests the performance of the montioring process when all of the SubtaskMaster instances have - * completed. - * - * @throws InterruptedException - */ - @Test - public void testMonitoringSubtaskMastersDone() - throws ConfigurationException, IllegalStateException, IOException, InterruptedException { - - when(subtaskServer.isListenerRunning()).thenReturn(true); - - // Initialize the instance. - computeNodeMaster.initialize(); - - // Replace the allPermitsAvailable() method with a mockery. - doReturn(true).when(computeNodeMaster).allPermitsAvailable(); - - // Run the monitoring process once. - computeNodeMaster.run(); - - // The countdown latch should no longer be waiting. - assertEquals(0, computeNodeMaster.getCountDownLatchCount()); + assertTrue(TimestampFile.timestamp(taskDir.toFile(), + TimestampFile.Event.ARRIVE_COMPUTE_NODES) > 0); + assertTrue(TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.START) > 0); } @Test @@ -397,28 +171,19 @@ public void testFinish() // If the subtasks aren't all done, then all we should get is a timestamp file. computeNodeMaster.finish(); - // The state file on disk should now be PROCESSING - assertEquals(StateFile.State.PROCESSING, stateFile.newStateFileFromDiskFile().getState()); - // The job finish timestamp should be present. - assertTrue( - TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.PBS_JOB_FINISH) > 0); + assertTrue(TimestampFile.timestamp(taskDir.toFile(), TimestampFile.Event.FINISH) > 0); // Mark subtasks as complete or failed. new AlgorithmStateFiles(taskDir.resolve("st-0").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); new AlgorithmStateFiles(taskDir.resolve("st-1").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); new AlgorithmStateFiles(taskDir.resolve("st-2").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); new AlgorithmStateFiles(taskDir.resolve("st-3").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); new AlgorithmStateFiles(taskDir.resolve("st-4").toFile()) - .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - - // This time, the state file should be updated. - computeNodeMaster.finish(); - - assertEquals(StateFile.State.COMPLETE, stateFile.newStateFileFromDiskFile().getState()); + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); } } diff --git a/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java index dbf8181..7da167f 100644 --- a/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java +++ b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java @@ -200,7 +200,7 @@ public void setup() throws IOException { .pipelineDefinitionNodeOperations(); List uows = PipelineExecutor.generateUnitsOfWork(uowGenerator, pipelineInstanceNode); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uows.get(0)); + Mockito.when(pipelineTask.getUnitOfWork()).thenReturn(uows.get(0)); // Construct mocked DatastoreFileManager. datastoreFileManager = Mockito.mock(DatastoreFileManager.class); diff --git a/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java b/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java index 3cff36b..5a1b2e7 100644 --- a/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java +++ b/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java @@ -36,15 +36,16 @@ import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineInputsSample; import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineOutputsSample1; import gov.nasa.ziggy.data.management.DatastoreProducerConsumerOperations; -import gov.nasa.ziggy.module.remote.TimestampFile.Event; +import gov.nasa.ziggy.module.TimestampFile.Event; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.database.DatabaseService; @@ -68,6 +69,7 @@ public class ExternalProcessPipelineModuleTest { private DatastoreFileManager datastoreFileManager; private DatastoreProducerConsumerOperations datastoreProducerConsumerOperations; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; @Rule public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -102,6 +104,7 @@ public void setup() { when(pipelineTaskOperations .pipelineModuleDefinition(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(pipelineModuleDefinition); + pipelineTaskDataOperations = Mockito.mock(PipelineTaskDataOperations.class); when(pipelineTask.taskBaseName()).thenReturn("50-100-test"); when(pipelineModuleDefinition.getInputsClass()) @@ -137,6 +140,7 @@ private void configurePipelineModule(RunMode runMode) { doReturn(datastoreProducerConsumerOperations).when(pipelineModule) .datastoreProducerConsumerOperations(); doReturn(pipelineTaskOperations).when(pipelineModule).pipelineTaskOperations(); + doReturn(pipelineTaskDataOperations).when(pipelineModule).pipelineTaskDataOperations(); doReturn(pipelineTask).when(pipelineModule).pipelineTask(); doReturn(taskConfiguration).when(pipelineModule).taskConfiguration(); doReturn(datastoreFileManager).when(pipelineModule).datastoreFileManager(); @@ -190,7 +194,7 @@ public void testExceptionFinalStep() { */ @Test public void testConstructor() { - assertEquals(100L, pipelineModule.taskId()); + assertEquals(100L, pipelineModule.pipelineTask().getId().longValue()); assertEquals(50L, pipelineModule.instanceId()); assertNotNull(pipelineModule.algorithmManager()); assertTrue(pipelineModule.pipelineInputs() instanceof PipelineInputsSample); @@ -207,8 +211,7 @@ public void testProcessMarshalingLocal() throws Exception { doReturn(ProcessingStep.MARSHALING).when(pipelineModule).currentProcessingStep(); pipelineModule.marshalingTaskAction(); - verify(pipelineModule).copyFilesToTaskDirectory(eq(taskConfiguration), - eq(taskDir)); + verify(pipelineModule).copyFilesToTaskDirectory(eq(taskConfiguration), eq(taskDir)); verify(taskConfiguration).serialize(eq(taskDir)); verify(pipelineModule).incrementProcessingStep(); assertFalse(pipelineModule.getDoneLooping()); @@ -226,8 +229,7 @@ public void testProcessMarshalingNoInputs() throws Exception { when(taskConfiguration.getSubtaskCount()).thenReturn(0); pipelineModule.marshalingTaskAction(); - verify(pipelineModule).copyFilesToTaskDirectory(eq(taskConfiguration), - eq(taskDir)); + verify(pipelineModule).copyFilesToTaskDirectory(eq(taskConfiguration), eq(taskDir)); verify(taskConfiguration, never()).serialize(eq(taskDir)); verify(pipelineModule, never()).incrementProcessingStep(); assertTrue(pipelineModule.getDoneLooping()); @@ -289,7 +291,6 @@ public void testProcessStoring() { when(failureSummary.isAllTasksFailed()).thenReturn(false); doReturn(0L).when(pipelineModule) .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); // the local version performs relatively limited activities @@ -655,7 +656,7 @@ public void testHaltStoring() { PipelineException exception = assertThrows(PipelineException.class, () -> pipelineModule.processingMainLoop()); - assertEquals("Unable to persist due to sub-task failures", exception.getMessage()); + assertEquals("Unable to persist due to subtask failures", exception.getMessage()); verify(pipelineModule, never()).marshalingTaskAction(); verify(pipelineModule, never()).submittingTaskAction(); verify(pipelineModule, never()).queuedTaskAction(); @@ -716,7 +717,7 @@ public void testRestartFromBeginning() { verify(pipelineModule, never()).executingTaskAction(); verify(pipelineModule, never()).waitingToStoreTaskAction(); verify(pipelineModule, never()).storingTaskAction(); - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), eq(ProcessingStep.SUBMITTING)); } @@ -730,7 +731,7 @@ public void testResubmitLocalTask() { ProcessingStep.STORING, ProcessingStep.STORING).when(pipelineModule) .currentProcessingStep(); pipelineModule.processTask(); - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), eq(ProcessingStep.SUBMITTING)); assertFalse(pipelineModule.isProcessingSuccessful()); verify(pipelineModule).processingMainLoop(); @@ -740,7 +741,8 @@ public void testResubmitLocalTask() { verify(pipelineModule, never()).executingTaskAction(); verify(pipelineModule, never()).waitingToStoreTaskAction(); verify(pipelineModule, never()).storingTaskAction(); - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), any(ProcessingStep.class)); + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), + any(ProcessingStep.class)); } /** Tests a resubmit for a remote execution task. */ @@ -754,7 +756,7 @@ public void testResubmitRemoteTask() { .currentProcessingStep(); when(taskAlgorithmLifecycle.isRemote()).thenReturn(true); pipelineModule.processTask(); - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), eq(ProcessingStep.SUBMITTING)); assertFalse(pipelineModule.isProcessingSuccessful()); verify(pipelineModule).processingMainLoop(); @@ -766,21 +768,6 @@ public void testResubmitRemoteTask() { verify(pipelineModule, never()).storingTaskAction(); } - @Test - public void testResumeMonitoring() { - configurePipelineModule(RunMode.RESUME_MONITORING); - doReturn(ProcessingStep.EXECUTING, ProcessingStep.EXECUTING, - ProcessingStep.WAITING_TO_STORE, ProcessingStep.WAITING_TO_STORE, - ProcessingStep.STORING, ProcessingStep.STORING).when(pipelineModule) - .currentProcessingStep(); - pipelineModule.processTask(); - assertFalse(pipelineModule.isProcessingSuccessful()); - verify(algorithmExecutor).resumeMonitoring(); - verify(pipelineModule, never()).processingMainLoop(); - verify(pipelineModule, never()).incrementProcessingStep(); - verify(pipelineModule, never()).currentProcessingStep(); - } - /** Test resumption of the marshaling step. */ @Test public void testResumeMarshaling() { @@ -802,7 +789,7 @@ public void testResumeMarshaling() { verify(pipelineModule, never()).executingTaskAction(); verify(pipelineModule, never()).waitingToStoreTaskAction(); verify(pipelineModule, never()).storingTaskAction(); - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), eq(ProcessingStep.SUBMITTING)); } @@ -823,7 +810,7 @@ public void testResumeAlgorithmExecuting() { verify(pipelineModule).executingTaskAction(); verify(pipelineModule, never()).waitingToStoreTaskAction(); verify(pipelineModule, never()).storingTaskAction(); - verify(pipelineTaskOperations, never()).updateProcessingStep(eq(100L), + verify(pipelineTaskDataOperations, never()).updateProcessingStep(eq(pipelineTask), eq(ProcessingStep.EXECUTING)); } @@ -847,7 +834,8 @@ public void testResumeAlgorithmComplete() { verify(pipelineModule).waitingToStoreTaskAction(); verify(pipelineModule).storingTaskAction(); // TODO Replace with another verification? - verify(pipelineTaskOperations).updateProcessingStep(eq(100L), eq(ProcessingStep.STORING)); + verify(pipelineTaskDataOperations).updateProcessingStep(eq(pipelineTask), + eq(ProcessingStep.STORING)); } @Test diff --git a/src/test/java/gov/nasa/ziggy/module/ProcessingFailureSummaryTest.java b/src/test/java/gov/nasa/ziggy/module/ProcessingFailureSummaryTest.java index 454f56a..192c513 100644 --- a/src/test/java/gov/nasa/ziggy/module/ProcessingFailureSummaryTest.java +++ b/src/test/java/gov/nasa/ziggy/module/ProcessingFailureSummaryTest.java @@ -30,12 +30,12 @@ public class ProcessingFailureSummaryTest { public void setup() { taskDir = directoryRule.directory().toFile(); - File subTaskDir = new File(taskDir, "st-0"); - subTaskDir.mkdirs(); - subTaskDir = new File(taskDir, "st-3"); - subTaskDir.mkdirs(); - subTaskDir = new File(taskDir, "st-200"); - subTaskDir.mkdirs(); + File subtaskDir = new File(taskDir, "st-0"); + subtaskDir.mkdirs(); + subtaskDir = new File(taskDir, "st-3"); + subtaskDir.mkdirs(); + subtaskDir = new File(taskDir, "st-200"); + subtaskDir.mkdirs(); } /** @@ -47,7 +47,7 @@ public void testNoErrors() { ProcessingFailureSummary p = new ProcessingFailureSummary(moduleName, taskDir); assertTrue(p.isAllTasksSucceeded()); assertFalse(p.isAllTasksFailed()); - assertEquals(0, p.getFailedSubTaskDirs().size()); + assertEquals(0, p.getFailedSubtaskDirs().size()); } /** @@ -62,13 +62,13 @@ public void testSomeErrors() throws IOException { ProcessingFailureSummary p = new ProcessingFailureSummary(moduleName, taskDir); assertFalse(p.isAllTasksSucceeded()); assertFalse(p.isAllTasksFailed()); - assertEquals(2, p.getFailedSubTaskDirs().size()); - assertEquals("st-3", p.getFailedSubTaskDirs().get(0)); - assertEquals("st-200", p.getFailedSubTaskDirs().get(1)); + assertEquals(2, p.getFailedSubtaskDirs().size()); + assertEquals("st-3", p.getFailedSubtaskDirs().get(0)); + assertEquals("st-200", p.getFailedSubtaskDirs().get(1)); } /** - * Test case with all sub-tasks errored. + * Test case with all subtasks errored. * * @throws IOException */ @@ -80,9 +80,9 @@ public void testAllErrored() throws IOException { ProcessingFailureSummary p = new ProcessingFailureSummary(moduleName, taskDir); assertFalse(p.isAllTasksSucceeded()); assertTrue(p.isAllTasksFailed()); - assertEquals(3, p.getFailedSubTaskDirs().size()); - assertEquals("st-0", p.getFailedSubTaskDirs().get(0)); - assertEquals("st-3", p.getFailedSubTaskDirs().get(1)); - assertEquals("st-200", p.getFailedSubTaskDirs().get(2)); + assertEquals(3, p.getFailedSubtaskDirs().size()); + assertEquals("st-0", p.getFailedSubtaskDirs().get(0)); + assertEquals("st-3", p.getFailedSubtaskDirs().get(1)); + assertEquals("st-200", p.getFailedSubtaskDirs().get(2)); } } diff --git a/src/test/java/gov/nasa/ziggy/module/StateFileTest.java b/src/test/java/gov/nasa/ziggy/module/StateFileTest.java deleted file mode 100644 index 98bc78e..0000000 --- a/src/test/java/gov/nasa/ziggy/module/StateFileTest.java +++ /dev/null @@ -1,311 +0,0 @@ - -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.services.config.PropertyName.RESULTS_DIR; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.fail; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; - -import org.apache.commons.io.FileUtils; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.RuleChain; -import org.mockito.Mockito; -import org.mockito.MockitoAnnotations; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.StateFile.State; -import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.module.remote.RemoteNodeDescriptor; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; -import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; - -/** - * Implements unit tests for {@link StateFile}. - */ -public class StateFileTest { - - private static final String QUEUE_NAME = "42424242"; - private static final String REMOTE_GROUP = "424242"; - private static final RemoteNodeDescriptor ARCHITECTURE = RemoteNodeDescriptor.SANDY_BRIDGE; - private static final int PFE_ARRIVAL_TIME_MILLIS = 45; - private static final int PBS_SUBMIT_TIME_MILLIS = 46; - private static final int MIN_CORES_PER_NODE = 3; - private static final int MIN_GIGS_PER_NODE = 2; - private static final int ACTIVE_CORES_PER_NODE = 1; - private static final int REQUESTED_NODE_COUNT = 2; - private static final double GIGS_PER_SUBTASK = 1.5; - public static final String REQUESTED_WALL_TIME = "4:30:00"; - public static final String EXECUTABLE_NAME = "executable2"; - - private static File workingDirectory; - - private static final String[] NAMES = { StateFile.PREFIX + "1.2.foo.INITIALIZED_1-0-0", - StateFile.PREFIX + "1.2.foo.INITIALIZED_1-0-1", - StateFile.PREFIX + "1.2.foo.INITIALIZED_1-1-0", - StateFile.PREFIX + "1.2.foo.INITIALIZED_2-0-0", StateFile.PREFIX + "1.2.foo.QUEUED_1-0-0", - StateFile.PREFIX + "1.2.foo.INITIALIZED_1-0-0", - StateFile.PREFIX + "1.3.foo.INITIALIZED_1-0-0", - StateFile.PREFIX + "2.2.foo.INITIALIZED_1-0-0", - StateFile.PREFIX + "1.2.foo.INITIALIZED_1-0-0", }; - - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - public ZiggyPropertyRule resultsDirPropertyRule = new ZiggyPropertyRule(RESULTS_DIR, - directoryRule); - - @Rule - public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) - .around(resultsDirPropertyRule); - - @Before - public void setUp() throws IOException { - Files.createDirectories(DirectoryProperties.stateFilesDir()); - workingDirectory = DirectoryProperties.stateFilesDir().toFile(); - } - - @Test - public void testConstructors() { - StateFile stateFile = createStateFile(); - testStateFileProperties(new StateFile(stateFile)); - - MockitoAnnotations.openMocks(this); - PipelineTask pipelineTask = createPipelineTask(); - PbsParameters pbsParameters = createPbsParameters(); - - stateFile = StateFile.generateStateFile(pipelineTask, pbsParameters, 0); - stateFile.setPbsSubmitTimeMillis(PBS_SUBMIT_TIME_MILLIS); - stateFile.setPfeArrivalTimeMillis(PFE_ARRIVAL_TIME_MILLIS); - stateFile.setRemoteGroup(REMOTE_GROUP); - testStateFileProperties(stateFile); - } - - private PipelineTask createPipelineTask() { - PipelineModuleDefinition moduleDefinition = new PipelineModuleDefinition("any"); - PipelineDefinitionNode pipelineDefinitionNode = new PipelineDefinitionNode("dummy", - moduleDefinition.getName()); - moduleDefinition.setExecutableName(EXECUTABLE_NAME); - PipelineInstance instance = new PipelineInstance(); - instance.setId(42L); - - PipelineTask task = Mockito.spy(new PipelineTask(instance, - new PipelineInstanceNode(pipelineDefinitionNode, moduleDefinition))); - Mockito.doReturn(43L).when(task).getId(); - - return task; - } - - private PbsParameters createPbsParameters() { - PbsParameters pbsParameters = new PbsParameters(); - - pbsParameters.setActiveCoresPerNode(ACTIVE_CORES_PER_NODE); - pbsParameters.setArchitecture(ARCHITECTURE); - pbsParameters.setMinCoresPerNode(MIN_CORES_PER_NODE); - pbsParameters.setMinGigsPerNode(MIN_GIGS_PER_NODE); - pbsParameters.setQueueName(QUEUE_NAME); - pbsParameters.setRemoteGroup(REMOTE_GROUP); - pbsParameters.setRequestedNodeCount(REQUESTED_NODE_COUNT); - pbsParameters.setRequestedWallTime(REQUESTED_WALL_TIME); - pbsParameters.setGigsPerSubtask(GIGS_PER_SUBTASK); - - return pbsParameters; - } - - @Test - public void testEqualsAndHash() { - for (String name1 : NAMES) { - StateFile a = new StateFile(name1); - assertFalse(a.equals(null)); - - for (String name2 : NAMES) { - StateFile b = new StateFile(name2); - if (!name1.equals(name2)) { - assertFalse(a.equals(b)); - assertFalse(b.equals(a)); - } else { - assertEquals(a, b); - assertEquals(b, a); - assertEquals(a.hashCode(), b.hashCode()); - - b = new StateFile(a.getModuleName(), a.getPipelineInstanceId(), - a.getPipelineTaskId()); - b.setState(a.getState()); - b.setNumTotal(a.getNumTotal()); - b.setNumComplete(a.getNumComplete()); - b.setNumFailed(a.getNumFailed()); - assertEquals(a, b); - assertEquals(b, a); - } - } - } - } - - @Test - public void testTaskDirName() { - StateFile a = new StateFile(StateFile.PREFIX + "1.2.hello.QUEUED_3-1-2"); - assertEquals("1-2-hello", a.taskDirName()); - } - - @Test - public void testName() { - StateFile stateFile = new StateFile(NAMES[0]); - assertEquals(StateFile.PREFIX + "1.2.foo.INITIALIZED_1-0-0", stateFile.name()); - assertEquals(StateFile.PREFIX + "1.2.foo", stateFile.invariantPart()); - } - - @Test - public void testState() { - StateFile stateFile = new StateFile(NAMES[0]); - stateFile.setState(State.CLOSED); - assertEquals(true, stateFile.isDone()); - assertEquals(false, stateFile.isRunning()); - assertEquals(false, stateFile.isQueued()); - stateFile.setState(State.COMPLETE); - assertEquals(true, stateFile.isDone()); - assertEquals(false, stateFile.isRunning()); - assertEquals(false, stateFile.isQueued()); - stateFile.setState(State.INITIALIZED); - assertEquals(false, stateFile.isDone()); - assertEquals(false, stateFile.isRunning()); - assertEquals(false, stateFile.isQueued()); - stateFile.setState(State.PROCESSING); - assertEquals(false, stateFile.isDone()); - assertEquals(true, stateFile.isRunning()); - assertEquals(false, stateFile.isQueued()); - stateFile.setState(State.QUEUED); - assertEquals(false, stateFile.isDone()); - assertEquals(false, stateFile.isRunning()); - assertEquals(true, stateFile.isQueued()); - stateFile.setState(State.SUBMITTED); - assertEquals(false, stateFile.isDone()); - assertEquals(false, stateFile.isRunning()); - assertEquals(false, stateFile.isQueued()); - } - - @Test - public void testDefaultPropertyValues() { - StateFile stateFile = new StateFile(NAMES[0]); - assertEquals(StateFile.DEFAULT_REMOTE_NODE_ARCHITECTURE, - stateFile.getRemoteNodeArchitecture()); - assertEquals(StateFile.DEFAULT_WALL_TIME, stateFile.getRequestedWallTime()); - assertEquals(StateFile.INVALID_STRING, stateFile.getRemoteGroup()); - assertEquals(StateFile.INVALID_STRING, stateFile.getQueueName()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getRequestedNodeCount()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getActiveCoresPerNode()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getMinCoresPerNode()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getMinGigsPerNode(), 1e-3); - - assertEquals(StateFile.INVALID_VALUE, stateFile.getPbsSubmitTimeMillis()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getPfeArrivalTimeMillis()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getGigsPerSubtask(), 1e-9); - assertEquals(StateFile.INVALID_STRING, stateFile.getExecutableName()); - } - - @Test - public void testUpdatedPropertyValues() { - StateFile stateFile = createStateFile(); - testStateFileProperties(stateFile); - } - - @Test - public void testFileIO() { - StateFile stateFile = createStateFile(); - - // Test argument error handling. - try { - FileUtils.deleteDirectory(workingDirectory); - } catch (IOException e) { - fail("Unexpected exception " + e); - } - - // Nominal write and read. - workingDirectory.mkdirs(); - try { - stateFile.persist(); - } catch (Exception e) { - fail("Unexpected exception " + e); - } - - try { - StateFile newStateFile = stateFile.newStateFileFromDiskFile(); - checkEqual(stateFile, newStateFile); - } catch (Exception e) { - fail("Unexpected exception " + e); - } - - // Update the state - StateFile newStateFile = new StateFile(stateFile); - newStateFile.setState(State.SUBMITTED); - StateFile.updateStateFile(stateFile, newStateFile); - try { - stateFile = newStateFile.newStateFileFromDiskFile(); - checkEqual(stateFile, newStateFile); - } catch (Exception e) { - fail("Unexpected exception " + e); - } - } - - private void checkEqual(StateFile stateFile, StateFile newStateFile) { - assertEquals(stateFile, newStateFile); - assertEquals(stateFile.hashCode(), newStateFile.hashCode()); - - // Check the rest of the properties that aren't in the equals method. - assertEquals(stateFile.getRemoteNodeArchitecture(), - newStateFile.getRemoteNodeArchitecture()); - assertEquals(stateFile.getRemoteGroup(), newStateFile.getRemoteGroup()); - assertEquals(stateFile.getQueueName(), newStateFile.getQueueName()); - assertEquals(stateFile.getRequestedWallTime(), newStateFile.getRequestedWallTime()); - assertEquals(stateFile.getActiveCoresPerNode(), newStateFile.getActiveCoresPerNode()); - assertEquals(stateFile.getRequestedNodeCount(), newStateFile.getRequestedNodeCount()); - assertEquals(stateFile.getMinCoresPerNode(), newStateFile.getMinCoresPerNode()); - assertEquals(stateFile.getMinGigsPerNode(), newStateFile.getMinGigsPerNode(), 1e-3); - - assertEquals(stateFile.getPbsSubmitTimeMillis(), newStateFile.getPbsSubmitTimeMillis()); - assertEquals(stateFile.getPfeArrivalTimeMillis(), newStateFile.getPfeArrivalTimeMillis()); - } - - private StateFile createStateFile() { - StateFile stateFile = new StateFile(NAMES[0]); - stateFile.setRemoteNodeArchitecture(ARCHITECTURE.getNodeName()); - stateFile.setRequestedWallTime(REQUESTED_WALL_TIME); - stateFile.setRemoteGroup(REMOTE_GROUP); - stateFile.setQueueName(QUEUE_NAME); - stateFile.setMinCoresPerNode(MIN_CORES_PER_NODE); - stateFile.setMinGigsPerNode(MIN_GIGS_PER_NODE); - stateFile.setActiveCoresPerNode(ACTIVE_CORES_PER_NODE); - stateFile.setRequestedNodeCount(REQUESTED_NODE_COUNT); - stateFile.setGigsPerSubtask(GIGS_PER_SUBTASK); - - stateFile.setPbsSubmitTimeMillis(PBS_SUBMIT_TIME_MILLIS); - stateFile.setPfeArrivalTimeMillis(PFE_ARRIVAL_TIME_MILLIS); - stateFile.setExecutableName(EXECUTABLE_NAME); - - return stateFile; - } - - private void testStateFileProperties(StateFile stateFile) { - assertEquals(ARCHITECTURE.getNodeName(), stateFile.getRemoteNodeArchitecture()); - assertEquals(REQUESTED_WALL_TIME, stateFile.getRequestedWallTime()); - assertEquals(REMOTE_GROUP, stateFile.getRemoteGroup()); - assertEquals(QUEUE_NAME, stateFile.getQueueName()); - assertEquals(MIN_CORES_PER_NODE, stateFile.getMinCoresPerNode()); - assertEquals(MIN_GIGS_PER_NODE, stateFile.getMinGigsPerNode(), 1e-3); - assertEquals(ACTIVE_CORES_PER_NODE, stateFile.getActiveCoresPerNode()); - assertEquals(REQUESTED_NODE_COUNT, stateFile.getRequestedNodeCount()); - assertEquals(GIGS_PER_SUBTASK, stateFile.getGigsPerSubtask(), 1e-9); - - assertEquals(PBS_SUBMIT_TIME_MILLIS, stateFile.getPbsSubmitTimeMillis()); - assertEquals(PFE_ARRIVAL_TIME_MILLIS, stateFile.getPfeArrivalTimeMillis()); - assertEquals(EXECUTABLE_NAME, stateFile.getExecutableName()); - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskClientTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskClientTest.java index c5059e1..21a7523 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskClientTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskClientTest.java @@ -14,6 +14,7 @@ import org.junit.After; import org.junit.Before; import org.junit.Test; +import org.mockito.Mockito; import gov.nasa.ziggy.TestEventDetector; import gov.nasa.ziggy.module.SubtaskServer.Request; @@ -37,7 +38,8 @@ public class SubtaskClientTest { @Before public void setUp() { - subtaskClient = new SubtaskClient(); + subtaskClient = Mockito.spy(SubtaskClient.class); + Mockito.when(subtaskClient.tryAgainWaitTimeMillis()).thenReturn(0L); threadPool = Executors.newSingleThreadExecutor(); requestQueue = SubtaskServer.getRequestQueue(); } diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskDirectoryIteratorTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskDirectoryIteratorTest.java index 6c8fa02..4bf7b4a 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskDirectoryIteratorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskDirectoryIteratorTest.java @@ -26,15 +26,15 @@ public class SubtaskDirectoryIteratorTest { @Test public void test() throws IOException { File taskDir = Files.createTempDir(); - File subTaskDir = new File(taskDir, "st-0"); - subTaskDir.mkdirs(); - subTaskDir = new File(taskDir, "st-3"); - subTaskDir.mkdirs(); - subTaskDir = new File(taskDir, "st-200"); - subTaskDir.mkdirs(); + File subtaskDir = new File(taskDir, "st-0"); + subtaskDir.mkdirs(); + subtaskDir = new File(taskDir, "st-3"); + subtaskDir.mkdirs(); + subtaskDir = new File(taskDir, "st-200"); + subtaskDir.mkdirs(); SubtaskDirectoryIterator it = new SubtaskDirectoryIterator(taskDir); - assertEquals(it.numSubTasks(), 3); + assertEquals(it.numSubtasks(), 3); assertEquals(-1, it.getCurrentIndex()); assertTrue(it.hasNext()); GroupSubtaskDirectory p = it.next(); diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java index dcfd66a..2fa6061 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java @@ -12,7 +12,6 @@ import java.io.File; import java.io.IOException; import java.io.Writer; -import java.nio.file.Files; import java.util.ArrayList; import java.util.List; @@ -29,7 +28,6 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineInputsSample; -import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.process.ExternalProcess; import gov.nasa.ziggy.util.os.OperatingSystemType; @@ -45,7 +43,7 @@ public class SubtaskExecutorTest { // at the present time. private File rootDir; private File taskDir; - private File subTaskDir; + private File subtaskDir; private SubtaskExecutor externalProcessExecutor; private ExternalProcess externalProcess; private TaskConfiguration taskConfigurationManager = new TaskConfiguration(); @@ -87,8 +85,8 @@ public void setup() throws IOException, ConfigurationException { rootDir = directoryRule.directory().toFile(); taskDir = new File(rootDir, "10-20-pa"); - subTaskDir = new File(taskDir, "st-0"); - subTaskDir.mkdirs(); + subtaskDir = new File(taskDir, "st-0"); + subtaskDir.mkdirs(); buildDir = new File(pipelineHomeDirPropertyRule.getValue()); binDir = new File(buildDir, "bin"); binDir.mkdirs(); @@ -96,14 +94,6 @@ public void setup() throws IOException, ConfigurationException { paFile.createNewFile(); new File(resultsDirPropertyRule.getValue()).mkdirs(); - - // Create the state file directory - Files.createDirectories(DirectoryProperties.stateFilesDir()); - - // Create the state file - StateFile stateFile = new StateFile("pa", 10, 20); - stateFile.setPfeArrivalTimeMillis(System.currentTimeMillis()); - stateFile.persist(); } @Test @@ -143,7 +133,7 @@ public void testConstructor() throws IOException { } // If this is a Mac, test the binaryFile + ".app" functionality - if (OperatingSystemType.getInstance() == OperatingSystemType.MAC_OS_X) { + if (OperatingSystemType.newInstance() == OperatingSystemType.MAC_OS_X) { File calFile = new File(binDir, "cal.app"); calFile = new File(calFile, "Contents"); calFile = new File(calFile, "MacOS"); @@ -193,7 +183,7 @@ public void testRunInputsOutputsCommand() throws ExecuteException, IOException { Mockito.when(externalProcess.execute()).thenReturn(0); int retCode = externalProcessExecutor.runInputsOutputsCommand(PipelineInputsSample.class); CommandLine commandLine = externalProcessExecutor.commandLine(); - Mockito.verify(externalProcess).setWorkingDirectory(subTaskDir); + Mockito.verify(externalProcess).setWorkingDirectory(subtaskDir); String cmdString = commandLine.toString(); String expectedCommandString = """ [/path/to/ziggy/build/bin/ziggy, --verbose,\s\ @@ -234,7 +224,7 @@ public void testInputsErrorSetsErrorStatus() throws Exception { .runInputsOutputsCommand(PipelineInputsSample.class); int retCode = externalProcessExecutor.execAlgorithmInternal(); assertEquals(1, retCode); - assertTrue(new File(subTaskDir, ".FAILED").exists()); + assertTrue(new File(subtaskDir, ".FAILED").exists()); Mockito.verify(externalProcessExecutor, Mockito.never()) .runCommandline(ArgumentMatchers.anyList(), ArgumentMatchers.any(String.class)); } @@ -250,7 +240,7 @@ public void testRunCommandLine() throws Exception { cmdLineArgs.add("dummyArg2"); int retCode = externalProcessExecutor.runCommandline(cmdLineArgs, "a"); CommandLine commandLine = externalProcessExecutor.commandLine(); - Mockito.verify(externalProcess).setWorkingDirectory(subTaskDir); + Mockito.verify(externalProcess).setWorkingDirectory(subtaskDir); Mockito.verify(externalProcess).setCommandLine(commandLine); Mockito.verify(externalProcess).setEnvironment(ArgumentMatchers.anyMap()); String cmdString = commandLine.toString(); diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java index 0464c98..35f7212 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java @@ -12,7 +12,7 @@ import java.io.File; import java.io.IOException; import java.nio.file.Paths; -import java.util.concurrent.Semaphore; +import java.util.concurrent.CountDownLatch; import org.junit.After; import org.junit.Before; @@ -47,10 +47,10 @@ public class SubtaskMasterTest { private SubtaskMaster subtaskMaster; private SubtaskClient subtaskClient; private SubtaskExecutor subtaskExecutor; - private Semaphore completionCounter; + private CountDownLatch countdownLatch; private AlgorithmStateFiles algorithmStateFiles; - private final int THREAD_NUMBER = 5; + private final int THREAD_COUNT = 5; private final String NODE = "dummy"; private final String BINARY_NAME = "dummy"; private final String TASK_DIR = "dummy"; @@ -66,9 +66,8 @@ public void setUp() throws IOException, InterruptedException { algorithmStateFiles = mock(AlgorithmStateFiles.class); // Set up the object for test. - completionCounter = new Semaphore(THREAD_NUMBER); - completionCounter.acquire(); - subtaskMaster = spy(new SubtaskMaster(THREAD_NUMBER, NODE, completionCounter, BINARY_NAME, + countdownLatch = new CountDownLatch(THREAD_COUNT); + subtaskMaster = spy(new SubtaskMaster(THREAD_COUNT, NODE, countdownLatch, BINARY_NAME, DirectoryProperties.taskDataDir().resolve(TASK_DIR).toString(), TIMEOUT)); doReturn(subtaskClient).when(subtaskMaster).subtaskClient(); doReturn(subtaskExecutor).when(subtaskMaster).subtaskExecutor(ArgumentMatchers.anyInt()); @@ -103,7 +102,7 @@ public void testNormalExecution() throws InterruptedException, IOException { .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -130,7 +129,7 @@ public void testAlgorithmFailure() throws InterruptedException, IOException { .toFile()); verify(subtaskMaster).logException(ArgumentMatchers.eq(SUBTASK_INDEX), ArgumentMatchers.any(ModuleFatalProcessingException.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -143,7 +142,7 @@ public void testSubtaskAlreadyComplete() throws InterruptedException, IOExceptio standardSetUp(); // The subtask should have a prior algorithm state file, one that indicates completion. - when(algorithmStateFiles.subtaskStateExists()).thenReturn(true); + when(algorithmStateFiles.stateExists()).thenReturn(true); when(algorithmStateFiles.isComplete()).thenReturn(true); // Execute the run() method. @@ -159,7 +158,7 @@ public void testSubtaskAlreadyComplete() throws InterruptedException, IOExceptio .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -172,7 +171,7 @@ public void testSubtaskAlreadyFailed() throws InterruptedException, IOException standardSetUp(); // The subtask should have a prior algorithm state file, one that indicates completion. - when(algorithmStateFiles.subtaskStateExists()).thenReturn(true); + when(algorithmStateFiles.stateExists()).thenReturn(true); when(algorithmStateFiles.isFailed()).thenReturn(true); // Execute the run() method. @@ -188,7 +187,7 @@ public void testSubtaskAlreadyFailed() throws InterruptedException, IOException .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -204,7 +203,7 @@ public void testSubtaskAlreadyProcessing() throws InterruptedException, IOExcept standardSetUp(); // The subtask should have a prior algorithm state file, one that indicates completion. - when(algorithmStateFiles.subtaskStateExists()).thenReturn(true); + when(algorithmStateFiles.stateExists()).thenReturn(true); when(algorithmStateFiles.isProcessing()).thenReturn(true); // Execute the run() method. @@ -220,7 +219,7 @@ public void testSubtaskAlreadyProcessing() throws InterruptedException, IOExcept .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -257,7 +256,7 @@ public void testUnableToObtainFileLock() throws InterruptedException, IOExceptio .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -288,7 +287,7 @@ public void testIOException() throws InterruptedException, IOException { Paths.get(TASK_DIR, "st-" + SUBTASK_INDEX, TaskConfiguration.LOCK_FILE_NAME).toFile()); verify(subtaskMaster).logException(ArgumentMatchers.eq(SUBTASK_INDEX), ArgumentMatchers.any(PipelineException.class)); - assertEquals(5, completionCounter.availablePermits()); + assertEquals(4, countdownLatch.getCount()); } /** @@ -314,7 +313,7 @@ private void standardSetUp() throws InterruptedException, IOException { .toFile()); // The subtask should have no prior algorithm state file. - when(algorithmStateFiles.subtaskStateExists()).thenReturn(false); + when(algorithmStateFiles.stateExists()).thenReturn(false); when(algorithmStateFiles.isProcessing()).thenReturn(false); when(algorithmStateFiles.isFailed()).thenReturn(false); when(algorithmStateFiles.isComplete()).thenReturn(false); diff --git a/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java b/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java index 8760fd9..0f0bef8 100644 --- a/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java @@ -3,6 +3,8 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; import java.io.File; import java.io.IOException; @@ -10,30 +12,44 @@ import java.nio.file.Path; import java.util.ArrayList; import java.util.List; +import java.util.concurrent.CountDownLatch; import org.apache.commons.configuration2.ex.ConfigurationException; import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; +import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; +import gov.nasa.ziggy.pipeline.definition.database.PipelineOperationsTestUtils; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.messages.AllJobsFinishedMessage; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; +import gov.nasa.ziggy.services.messages.WorkerStatusMessage; /** * Unit tests for {@link TestMonitor} class. * * @author PT + * @author Bill Wohler */ public class TaskMonitorTest { private Path taskDir; - private Path stateFileDir; - private StateFile stateFile; private TaskMonitor taskMonitor; + private PipelineTask pipelineTask; private List algorithmStateFiles = new ArrayList<>(); + private AlgorithmStateFiles taskAlgorithmStateFiles; + private PipelineTaskDataOperations pipelineTaskDataOperations = new PipelineTaskDataOperations(); public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -43,15 +59,15 @@ public class TaskMonitorTest { @Rule public RuleChain ruleChain = RuleChain.outerRule(directoryRule).around(pipelineResultsRule); + @Rule + public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); + @Before public void setUp() throws IOException, ConfigurationException { taskDir = DirectoryProperties.taskDataDir().resolve("10-20-modulename"); - stateFileDir = DirectoryProperties.stateFilesDir(); Files.createDirectories(taskDir); - Files.createDirectories(stateFileDir); - Files.createFile(taskDir.resolve(StateFile.LOCK_FILE_NAME)); - + taskAlgorithmStateFiles = new AlgorithmStateFiles(taskDir.toFile()); // Create 6 subtask directories List subtaskDirectories = new ArrayList<>(); for (int subtask = 0; subtask < 6; subtask++) { @@ -61,129 +77,246 @@ public void setUp() throws IOException, ConfigurationException { algorithmStateFiles.add(new AlgorithmStateFiles(subtaskDir.toFile())); } - stateFile = StateFile.of(taskDir); - stateFile.setActiveCoresPerNode(1); - stateFile.setNumTotal(subtaskDirectories.size()); - stateFile.persist(); + // Set up the database + PipelineOperationsTestUtils testUtils = new PipelineOperationsTestUtils(); + testUtils.setUpSingleModulePipeline(); + pipelineTask = testUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, 6, -1, -1); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.QUEUED); - taskMonitor = new TaskMonitor(stateFile, taskDir.toFile()); + // And, finally, the TaskMonitor itself. + taskMonitor = new TaskMonitor(pipelineTask, taskDir.toFile(), 100L); + taskMonitor = Mockito.spy(taskMonitor); + Mockito.doReturn(0L).when(taskMonitor).fileSystemCheckIntervalMillis(); + Mockito.doReturn(0).when(taskMonitor).fileSystemChecksCount(); } @Test - public void testUpdateState() throws IOException { - stateFile = taskMonitor.getStateFile(); - assertEquals(StateFile.State.INITIALIZED, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); - - // Setting a subtask to PROCESSING should not affect the counts. - algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.SubtaskState.PROCESSING); - taskMonitor.updateState(); - assertEquals(StateFile.State.PROCESSING, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); - - // Setting a subtask to COMPLETED will affect the results. - algorithmStateFiles.get(1).updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - taskMonitor.updateState(); - assertEquals(StateFile.State.PROCESSING, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(1, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); - - // Setting a subtask to FAILED will affect the results. - algorithmStateFiles.get(2).updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - taskMonitor.updateState(); - assertEquals(StateFile.State.PROCESSING, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(1, stateFile.getNumComplete()); - assertEquals(1, stateFile.getNumFailed()); - } + public void testUpdate() { - @Test - public void testUpdateStateFileState() { - stateFile = taskMonitor.getStateFile(); - assertEquals(StateFile.State.INITIALIZED, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); - - stateFile.setState(StateFile.State.QUEUED); - stateFile.persist(); - stateFile = taskMonitor.getStateFile(); - assertEquals(StateFile.State.QUEUED, stateFile.getState()); + taskMonitor.update(); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // Update the task state and see that the database gets updated. + taskAlgorithmStateFiles.updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); + taskMonitor.update(); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // Update a subtask to PROCESSING, which should do nothing. + algorithmStateFiles.get(0) + .updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); + taskMonitor.update(); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // Update a subtask to completed, which will cause the database values to change. + algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + taskMonitor.update(); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 1, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // Set a subtask to failed, which will cause the database values to change. + algorithmStateFiles.get(1).updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); + taskMonitor.update(); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 1, 1, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // Get the rest of the subasks to either completed or failed. Note that it's not the + // TaskMonitor that advances the task state to WAITING_TO_STORE; that's the job of the + // AlgorithmMonitor, because it needs to disposition the task at the end of execution. + algorithmStateFiles.get(2).updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); + algorithmStateFiles.get(3).updateCurrentState(AlgorithmStateFiles.AlgorithmState.FAILED); + algorithmStateFiles.get(4).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + algorithmStateFiles.get(5).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + taskMonitor.update(); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 3, 3, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertTrue(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(1)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(1)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); } @Test - public void updateFromQueuedToProcessingState() { - stateFile = taskMonitor.getStateFile(); - assertEquals(StateFile.State.INITIALIZED, stateFile.getState()); - - // Set to QUEUED state. - stateFile.setState(StateFile.State.QUEUED); - stateFile.persist(); - stateFile = taskMonitor.getStateFile(); - assertEquals(StateFile.State.QUEUED, stateFile.getState()); - - // Update a subtask to completed. - algorithmStateFiles.get(1).updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - taskMonitor.updateState(); - assertEquals(StateFile.State.PROCESSING, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(1, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); + public void testWorkerStatusMessage() { + + taskAlgorithmStateFiles.updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); + algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // A final message from the wrong worker isn't interesting. + PipelineTask someOtherTask = spy(PipelineTask.class); + when(someOtherTask.getId()).thenReturn(1000L); + WorkerStatusMessage workerStatusMessage = new WorkerStatusMessage(1, "", + Long.toString(pipelineTask.getPipelineInstanceId()), someOtherTask, "", "", 0, true); + taskMonitor.handleWorkerStatusMessage(workerStatusMessage); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(0)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(0)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); + + // A non-final message from the right worker isn't interesting, either. + taskMonitor.handleWorkerStatusMessage(new WorkerStatusMessage(1, "", + Long.toString(pipelineTask.getPipelineInstanceId()), pipelineTask, "", "", 0, false)); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(0)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(0)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); + + // A final message from the worker isn't interesting if the task is running remotely. + pipelineTaskDataOperations.updateAlgorithmType(pipelineTask, AlgorithmType.REMOTE); + taskMonitor.handleWorkerStatusMessage(new WorkerStatusMessage(1, "", + Long.toString(pipelineTask.getPipelineInstanceId()), pipelineTask, "", "", 0, true)); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(0)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(0)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); + + // A final message from the right worker causes a final update. + pipelineTaskDataOperations.updateAlgorithmType(pipelineTask, AlgorithmType.LOCAL); + taskMonitor.handleWorkerStatusMessage(new WorkerStatusMessage(1, "", + Long.toString(pipelineTask.getPipelineInstanceId()), pipelineTask, "", "", 0, true)); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 1, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(1)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(1)).publishTaskProcessingCompleteMessage(null); } - /** - * Tests whether marking a state file as complete when it has subtasks that never got run moves - * those subtasks to being counted in the state file as failed. - * - * @throws IOException - */ @Test - public void testMarkStateFileCompleteWithSkippedTasks() throws IOException { - taskMonitor.markStateFileDone(); - assertEquals(StateFile.State.COMPLETE, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumComplete()); - assertEquals(6, stateFile.getNumFailed()); - StateFile diskStateFile = stateFile.newStateFileFromDiskFile(); - assertEquals(StateFile.State.COMPLETE, diskStateFile.getState()); - assertEquals(6, diskStateFile.getNumTotal()); - assertEquals(0, diskStateFile.getNumComplete()); - assertEquals(6, diskStateFile.getNumFailed()); + public void testAllJobsFinishedMessage() { + taskAlgorithmStateFiles.updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); + algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // A message for the wrong task isn't interesting. + PipelineTask someOtherTask = spy(PipelineTask.class); + when(someOtherTask.getId()).thenReturn(1000L); + AllJobsFinishedMessage allJobsFinishedMessage = new AllJobsFinishedMessage(someOtherTask); + taskMonitor.handleAllJobsFinishedMessage(allJobsFinishedMessage); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(0)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(0)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); + + // A message for the right task causes a final update. + allJobsFinishedMessage = new AllJobsFinishedMessage(pipelineTask); + taskMonitor.handleAllJobsFinishedMessage(allJobsFinishedMessage); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 1, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(1)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(1)).publishTaskProcessingCompleteMessage(null); } @Test - public void markStateFileComplete() throws IOException { - for (AlgorithmStateFiles algorithmStateFile : algorithmStateFiles) { - algorithmStateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - } - taskMonitor.updateState(); - taskMonitor.markStateFileDone(); - assertEquals(StateFile.State.COMPLETE, stateFile.getState()); - assertEquals(6, stateFile.getNumTotal()); - assertEquals(6, stateFile.getNumComplete()); - assertEquals(0, stateFile.getNumFailed()); - StateFile diskStateFile = stateFile.newStateFileFromDiskFile(); - assertEquals(StateFile.State.COMPLETE, diskStateFile.getState()); - assertEquals(6, diskStateFile.getNumTotal()); - assertEquals(6, diskStateFile.getNumComplete()); - assertEquals(0, diskStateFile.getNumFailed()); + public void testHaltTasksRequest() { + taskAlgorithmStateFiles.updateCurrentState(AlgorithmStateFiles.AlgorithmState.PROCESSING); + algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.AlgorithmState.COMPLETE); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + + // A message for the wrong task isn't interesting. + PipelineTask someOtherTask = spy(PipelineTask.class); + when(someOtherTask.getId()).thenReturn(1000L); + HaltTasksRequest request = new HaltTasksRequest(List.of(someOtherTask)); + taskMonitor.handleHaltTasksRequest(request); + assertEquals(ProcessingStep.QUEUED, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 0, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(0)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(0)) + .publishTaskProcessingCompleteMessage(ArgumentMatchers.any(CountDownLatch.class)); + + // A message for the right task causes a final update. + request = new HaltTasksRequest(List.of(pipelineTask)); + taskMonitor.handleHaltTasksRequest(request); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + TaskCountsTest.testSubtaskCounts(6, 1, 0, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + assertFalse(taskMonitor.allSubtasksProcessed()); + Mockito.verify(taskMonitor, Mockito.times(1)).shutdown(); + Mockito.verify(taskMonitor, Mockito.times(1)).publishTaskProcessingCompleteMessage(null); } @Test - public void testAllSubtasksProcessed() throws IOException { - assertFalse(taskMonitor.allSubtasksProcessed()); - for (AlgorithmStateFiles algorithmStateFile : algorithmStateFiles) { - algorithmStateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - } - taskMonitor.updateState(); - assertTrue(taskMonitor.allSubtasksProcessed()); - algorithmStateFiles.get(0).updateCurrentState(AlgorithmStateFiles.SubtaskState.FAILED); - taskMonitor.updateState(); - assertTrue(taskMonitor.allSubtasksProcessed()); + public void testCheckForFinishFile() { + Mockito.doReturn(10L).when(taskMonitor).fileSystemCheckIntervalMillis(); + Mockito.doReturn(1).when(taskMonitor).fileSystemChecksCount(); + + // A garden variety update with no finish file returns false. + taskMonitor.update(); + assertFalse(taskMonitor.isFinishFileDetected()); + + // A final update with no finish file returns false. + HaltTasksRequest request = new HaltTasksRequest(List.of(pipelineTask)); + taskMonitor.handleHaltTasksRequest(request); + assertFalse(taskMonitor.isFinishFileDetected()); + + // A final update with a finish file returns true. + taskMonitor.resetMonitoringEnabled(); + TimestampFile.create(taskDir.toFile(), TimestampFile.Event.FINISH); + taskMonitor.handleHaltTasksRequest(request); + assertTrue(taskMonitor.isFinishFileDetected()); + + // Even when there is a finish file, the garden variety update doesn't + // even look for it. + taskMonitor.resetFinishFileDetection(); + taskMonitor.resetMonitoringEnabled(); + taskMonitor.update(); + assertFalse(taskMonitor.isFinishFileDetected()); } } diff --git a/src/test/java/gov/nasa/ziggy/module/remote/TimestampFileTest.java b/src/test/java/gov/nasa/ziggy/module/TimestampFileTest.java similarity index 89% rename from src/test/java/gov/nasa/ziggy/module/remote/TimestampFileTest.java rename to src/test/java/gov/nasa/ziggy/module/TimestampFileTest.java index 7a5567a..9cef384 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/TimestampFileTest.java +++ b/src/test/java/gov/nasa/ziggy/module/TimestampFileTest.java @@ -1,4 +1,4 @@ -package gov.nasa.ziggy.module.remote; +package gov.nasa.ziggy.module; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; @@ -26,12 +26,11 @@ public class TimestampFileTest { @Test public void test() throws IOException { long timeMillis = System.currentTimeMillis(); - String expectedName = "QUEUED_PBS." + timeMillis; + String expectedName = "QUEUED." + timeMillis; File directory = directoryRule.directory().toFile(); - boolean success = TimestampFile.create(directory, TimestampFile.Event.QUEUED_PBS, - timeMillis); + boolean success = TimestampFile.create(directory, TimestampFile.Event.QUEUED, timeMillis); assertTrue("success", success); @@ -45,7 +44,7 @@ public void test() throws IOException { assertTrue("group can reaad", permissions.contains(PosixFilePermission.GROUP_READ)); assertTrue("other can reaad", permissions.contains(PosixFilePermission.OTHERS_READ)); - long actualTime = TimestampFile.timestamp(directory, TimestampFile.Event.QUEUED_PBS); + long actualTime = TimestampFile.timestamp(directory, TimestampFile.Event.QUEUED); assertEquals("timestamp", timeMillis, actualTime); } diff --git a/src/test/java/gov/nasa/ziggy/module/remote/PbsLogParserTest.java b/src/test/java/gov/nasa/ziggy/module/remote/PbsLogParserTest.java new file mode 100644 index 0000000..478e3c7 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/module/remote/PbsLogParserTest.java @@ -0,0 +1,69 @@ +package gov.nasa.ziggy.module.remote; + +import static gov.nasa.ziggy.ZiggyUnitTestUtils.TEST_DATA; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +import org.junit.Before; +import org.junit.Test; + +public class PbsLogParserTest { + + private static final Path PBS_LOG_FILE_DIR = TEST_DATA.resolve("PbsLogParser"); + private RemoteJobInformation job0Information; + private RemoteJobInformation job1Information; + private List remoteJobsInformation; + + @Before + public void setUp() { + job0Information = new RemoteJobInformation( + PBS_LOG_FILE_DIR.resolve("pbs-log-comment-and-status").toString(), "test1"); + job0Information.setJobId(1023L); + job1Information = new RemoteJobInformation( + PBS_LOG_FILE_DIR.resolve("pbs-log-status-no-comment.txt").toString(), "test2"); + job1Information.setJobId(1024L); + remoteJobsInformation = List.of(job0Information, job1Information); + } + + @Test + public void testExitCommentByJobId() { + Map exitCommentByJobId = new PbsLogParser() + .exitCommentByJobId(remoteJobsInformation); + assertNotNull(exitCommentByJobId.get(1023L)); + assertEquals("job killed: walltime 1818 exceeded limit 1800", + exitCommentByJobId.get(1023L)); + assertEquals(1, exitCommentByJobId.size()); + } + + @Test + public void testExitStatusByJobId() { + Map exitStatusByJobId = new PbsLogParser() + .exitStatusByJobId(remoteJobsInformation); + assertNotNull(exitStatusByJobId.get(1023L)); + assertEquals(271, exitStatusByJobId.get(1023L).intValue()); + assertNotNull(exitStatusByJobId.get(1024L)); + assertEquals(0, exitStatusByJobId.get(1024L).intValue()); + assertEquals(2, exitStatusByJobId.size()); + } + + @Test + public void testMissingPbsLog() { + RemoteJobInformation job2Information = new RemoteJobInformation("no-such-file", "test3"); + job2Information.setJobId(1025L); + Map exitStatusByJobId = new PbsLogParser() + .exitStatusByJobId(remoteJobsInformation); + assertNull(exitStatusByJobId.get(1025L)); + assertNotNull(exitStatusByJobId.get(1024L)); + assertNotNull(exitStatusByJobId.get(1023L)); + Map exitCommentByJobId = new PbsLogParser() + .exitCommentByJobId(remoteJobsInformation); + assertNull(exitCommentByJobId.get(1025L)); + assertNull(exitCommentByJobId.get(1024L)); + assertNotNull(exitCommentByJobId.get(1023L)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/module/remote/QstatMonitorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/QstatMonitorTest.java deleted file mode 100644 index 2683507..0000000 --- a/src/test/java/gov/nasa/ziggy/module/remote/QstatMonitorTest.java +++ /dev/null @@ -1,299 +0,0 @@ -package gov.nasa.ziggy.module.remote; - -import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_QUEUE_COMMAND_CLASS; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertSame; -import static org.junit.Assert.assertTrue; - -import java.util.Map; -import java.util.Set; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.mockito.Mockito; - -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.StateFile; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; - -/** - * Unit test class for QstatMonitor class, QstatQuery class, and the builder for the QstatMonitor - * class. - * - * @author PT - */ -public class QstatMonitorTest { - - private QueueCommandManager cmdManager; - - @Rule - public ZiggyPropertyRule queueCommandClassPropertyRule = new ZiggyPropertyRule( - REMOTE_QUEUE_COMMAND_CLASS, "gov.nasa.ziggy.module.remote.QueueCommandManagerForUnitTests"); - - @Before - public void setup() { - - // make sure that we have an instance of the command manager that we can use - cmdManager = Mockito.spy(QueueCommandManager.newInstance()); - Mockito.when(cmdManager.hostname()).thenReturn("host1"); - Mockito.when(cmdManager.user()).thenReturn("user"); - } - - /** - * Test all constructor syntaxes. Also exercises the getOwner(), getServerName(), and - * getQstatCommandManager() methods. - */ - @Test - public void testConstructors() { - - // 2-argument constructor - QstatMonitor monitor = new QstatMonitor("user", "server"); - assertEquals("user", monitor.getOwner()); - assertEquals("server", monitor.getServerName()); - assertTrue(monitor.getQstatCommandManager() instanceof QueueCommandManagerForUnitTests); - - // 1-argument constructor - monitor = new QstatMonitor(cmdManager); - assertEquals("user", monitor.getOwner()); - assertEquals("host1", monitor.getServerName()); - assertSame(cmdManager, monitor.getQstatCommandManager()); - } - - /** - * Tests addToMonitor methods. Also exercises the jobsInMonitor() method. - */ - @Test - public void testAddToMonitoring() { - StateFile stateFile = new StateFile("tps", 100, 200); - QstatMonitor monitor = new QstatMonitor(cmdManager); - monitor.addToMonitoring(stateFile); - Set jobsInMonitor = monitor.getJobsInMonitor(); - assertEquals(1, jobsInMonitor.size()); - assertTrue(jobsInMonitor.contains("100-200-tps")); - - PipelineTask task = Mockito.mock(PipelineTask.class); - Mockito.when(task.taskBaseName()).thenReturn(PipelineTask.taskBaseName(100L, 201L, "tps")); - - monitor.addToMonitoring(task); - jobsInMonitor = monitor.getJobsInMonitor(); - assertEquals(2, jobsInMonitor.size()); - assertTrue(jobsInMonitor.contains("100-200-tps")); - assertTrue(jobsInMonitor.contains("100-201-tps")); - } - - /** - * Tests the endMonitor method. - */ - @Test - public void testEndMonitoring() { - QstatMonitor monitor = new QstatMonitor(cmdManager); - monitor.addToMonitoring(new StateFile("tps", 100, 200)); - monitor.addToMonitoring(new StateFile("tps", 100, 201)); - monitor.endMonitoring(new StateFile("tps", 100, 200)); - Set jobsInMonitor = monitor.getJobsInMonitor(); - assertEquals(1, jobsInMonitor.size()); - assertTrue(jobsInMonitor.contains("100-201-tps")); - } - - /** - * Exercises the update method. Also exercises the jobId method, the isFinished method, the - * exitStatus method, and the exitComment method. - */ - @Test - public void testUpdate() { - QstatMonitor monitor = new QstatMonitor(cmdManager); - monitor.addToMonitoring(new StateFile("tps", 100, 200)); - monitor.addToMonitoring(new StateFile("tps", 100, 201)); - monitor.addToMonitoring(new StateFile("tps", 100, 202)); - - // mock the returns for each of the 3 tasks: note that the first task will - // return exactly 1 job, the second task will return 2 jobs (different server - // name), and the 3rd job will return nothing. - - String header1 = " Req'd Elap"; - String header2 = "JobID User Queue Jobname TSK Nds wallt S wallt Eff"; - String header3 = "-------------- ------- ----- ------------- --- --- ----- - ----- ----"; - - String qstat1 = "1234567.batch user low 100-200-tps 5 5 04:00 R 02:33 254%"; - String qstat2a = "1234587.batch user low 100-201-tps 5 5 04:00 R 02:33 254%"; - String qstat2b = "1234597.batch user low 100-201-tps 5 5 04:00 R 02:33 254%"; - - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-200-tps" }, - qstat1); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-201-tps" }, - qstat2a, qstat2b); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-202-tps" }, - (String[]) null); - - // mock the return of the server names - String[] jobOrOwner = { "Job:", "Job_Owner" }; - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567", jobOrOwner, - "Job: 1234567.batch.example.com", " Job_Owner = user@host1.example.com"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587 1234597", jobOrOwner, - "Job: 1234587.batch.example.com", " Job_Owner = user@host2.example.com", - "Job: 1234597.batch.example.com", " Job_Owner = user@host1.example.com"); - - monitor.update(); - - // set up the pipeline tasks so we can retrieve job IDs - - PipelineTask task1 = Mockito.mock(PipelineTask.class); - Mockito.when(task1.taskBaseName()).thenReturn(PipelineTask.taskBaseName(100L, 200L, "tps")); - - PipelineTask task2 = Mockito.mock(PipelineTask.class); - Mockito.when(task2.taskBaseName()).thenReturn(PipelineTask.taskBaseName(100L, 201L, "tps")); - - Mockito.when(task2.getId()).thenReturn(201L); - - Set task1Jobs = monitor.allIncompleteJobIds(task1); - Set task2Jobs = monitor.allIncompleteJobIds(task2); - assertEquals(1, task1Jobs.size()); - assertTrue(task1Jobs.contains(1234567L)); - assertEquals(1, task2Jobs.size()); - assertTrue(task2Jobs.contains(1234597L)); - - // now perform an update -- this allows the state of each job to be - // updated - - String qstat1Update = "1234567.batch user low 100-200-tps 5 5 04:00 F 02:33 254%"; - String qstat2Update = "1234597.batch user low 100-201-tps 5 5 04:00 R 02:34 254%"; - QueueCommandManagerTest.mockQstatCall(cmdManager, "-x 1234567 1234597", header1, header2, - header3, qstat1Update, qstat2Update); - - // before we perform the further update, neither task should be finished - assertFalse(monitor.isFinished(new StateFile("tps", 100, 200))); - assertFalse(monitor.isFinished(new StateFile("tps", 100, 201))); - monitor.update(); - - // now the first task should be finished but not the second - assertTrue(monitor.isFinished(new StateFile("tps", 100, 200))); - assertFalse(monitor.isFinished(new StateFile("tps", 100, 201))); - - // finally test the exit status retriever - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567", - new String[] { "Exit_status" }, "Exit_status = 0"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567", new String[] { "comment" }, - "comment = finished, okay?"); - Map exitStatusMap = monitor.exitStatus(new StateFile("tps", 100, 200)); - assertEquals(1, exitStatusMap.size()); - assertEquals(0, exitStatusMap.get(1234567L).intValue()); - Map exitCommentMap = monitor.exitComment(new StateFile("tps", 100, 200)); - assertEquals(1, exitCommentMap.size()); - assertEquals("finished, okay?", exitCommentMap.get(1234567L)); - } - - /** - * Exercises the update method for the use-case in which tasks produce multiple jobs. - */ - @Test - public void testUpdateMultipleJobsPerTask() { - QstatMonitor monitor = new QstatMonitor(cmdManager); - monitor.addToMonitoring(new StateFile("tps", 100, 200)); - monitor.addToMonitoring(new StateFile("tps", 100, 201)); - monitor.addToMonitoring(new StateFile("tps", 100, 202)); - - // mock the returns for each of the 3 tasks: note that the first task will - // return exactly 2 jobs, the second task will return 3 jobs (different server - // name for one of them), and the 3rd job will return nothing. - - String header1 = " Req'd Elap"; - String header2 = "JobID User Queue Jobname TSK Nds wallt S wallt Eff"; - String header3 = "-------------- ------- ----- ------------- --- --- ----- - ----- ----"; - - String qstat1a = "1234567.batch user low 100-200-tps.0 5 5 04:00 R 02:33 254%"; - String qstat1b = "1234568.batch user low 100-200-tps.1 5 5 04:00 R 02:33 254%"; - String qstat2a = "1234587.batch user low 100-201-tps.0 5 5 04:00 R 02:33 254%"; - String qstat2b = "1234597.batch user low 100-201-tps.1 5 5 04:00 R 02:33 254%"; - String qstat2c = "1234599.batch user low 100-201-tps 5 5 04:00 R 02:33 254%"; - - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-200-tps" }, - qstat1a, qstat1b); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-201-tps" }, - qstat2a, qstat2b, qstat2c); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-202-tps" }, - (String[]) null); - - // mock the return of the server names - String[] jobOrOwner = { "Job:", "Job_Owner" }; - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567 1234568", jobOrOwner, - "Job: 1234567.batch.example.com", " Job_Owner = user@host1.example.com", - "Job: 1234568.batch.example.com", " Job_Owner = user@host1.example.com"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587 1234597 1234599", jobOrOwner, - "Job: 1234587.batch.example.com", " Job_Owner = user@host1.example.com", - "Job: 1234597.batch.example.com", " Job_Owner = user@host1.example.com", - "Job: 1234599.batch.example.com", " Job_Owner = user@host2.example.com"); - - monitor.update(); - - // set up the pipeline tasks so we can retrieve job IDs - - PipelineTask task1 = Mockito.mock(PipelineTask.class); - Mockito.when(task1.taskBaseName()).thenReturn(PipelineTask.taskBaseName(100L, 200L, "tps")); - - PipelineTask task2 = Mockito.mock(PipelineTask.class); - Mockito.when(task2.taskBaseName()).thenReturn(PipelineTask.taskBaseName(100L, 201L, "tps")); - - Set task1Jobs = monitor.allIncompleteJobIds(task1); - Set task2Jobs = monitor.allIncompleteJobIds(task2); - assertEquals(2, task1Jobs.size()); - assertTrue(task1Jobs.contains(1234567L)); - assertTrue(task1Jobs.contains(1234568L)); - assertEquals(2, task2Jobs.size()); - assertTrue(task2Jobs.contains(1234587L)); - assertTrue(task2Jobs.contains(1234597L)); - - // now perform an update -- this allows the state of each job to be - // updated - - String qstat1aUpdate = "1234567.batch user low 100-200-tps.0 5 5 04:00 F 02:33 254%"; - String qstat1bUpdate = "1234568.batch user low 100-200-tps.1 5 5 04:00 F 02:33 254%"; - String qstat2aUpdate = "1234587.batch user low 100-201-tps.0 5 5 04:00 R 02:34 254%"; - String qstat2bUpdate = "1234597.batch user low 100-201-tps.1 5 5 04:00 F 02:34 254%"; - QueueCommandManagerTest.mockQstatCall(cmdManager, "-x 1234567 1234568 1234587 1234597", - header1, header2, header3, qstat1aUpdate, qstat1bUpdate, qstat2aUpdate, qstat2bUpdate); - - // before we perform the further update, neither task should be finished - assertFalse(monitor.isFinished(new StateFile("tps", 100, 200))); - assertFalse(monitor.isFinished(new StateFile("tps", 100, 201))); - monitor.update(); - - // now the first task should be finished but not the second - assertTrue(monitor.isFinished(new StateFile("tps", 100, 200))); - assertFalse(monitor.isFinished(new StateFile("tps", 100, 201))); - - // finally test the exit status retriever - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567", - new String[] { "Exit_status" }, "Exit_status = 0"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567", new String[] { "comment" }, - "comment = finished, okay?"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234568", - new String[] { "Exit_status" }, "Exit_status = 1"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234568", new String[] { "comment" }, - "comment = really finished"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587", - new String[] { "Exit_status" }, (String[]) null); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587", new String[] { "comment" }, - (String[]) null); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234597", - new String[] { "Exit_status" }, "Exit_status = 2"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234597", new String[] { "comment" }, - "comment = crashed and burned"); - Map exitStatusMap = monitor.exitStatus(new StateFile("tps", 100, 200)); - assertEquals(2, exitStatusMap.size()); - assertEquals(0, exitStatusMap.get(1234567L).intValue()); - assertEquals(1, exitStatusMap.get(1234568L).intValue()); - Map exitCommentMap = monitor.exitComment(new StateFile("tps", 100, 200)); - assertEquals(2, exitCommentMap.size()); - assertEquals("finished, okay?", exitCommentMap.get(1234567L)); - assertEquals("really finished", exitCommentMap.get(1234568L)); - - exitStatusMap = monitor.exitStatus(new StateFile("tps", 100, 201)); - assertEquals(1, exitStatusMap.size()); - assertEquals(2, exitStatusMap.get(1234597L).intValue()); - exitCommentMap = monitor.exitComment(new StateFile("tps", 100, 201)); - assertEquals(1, exitCommentMap.size()); - assertEquals("crashed and burned", exitCommentMap.get(1234597L)); - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/remote/QstatParserTest.java b/src/test/java/gov/nasa/ziggy/module/remote/QstatParserTest.java new file mode 100644 index 0000000..e349b6a --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/module/remote/QstatParserTest.java @@ -0,0 +1,119 @@ +package gov.nasa.ziggy.module.remote; + +import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_QUEUE_COMMAND_CLASS; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.util.List; +import java.util.Map; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +public class QstatParserTest { + + private QueueCommandManager cmdManager; + + @Rule + public ZiggyPropertyRule queueCommandClassPropertyRule = new ZiggyPropertyRule( + REMOTE_QUEUE_COMMAND_CLASS, "gov.nasa.ziggy.module.remote.QueueCommandManagerForUnitTests"); + private PipelineTask pipelineTask; + + @Before + public void setup() { + + // make sure that we have an instance of the command manager that we can use + cmdManager = Mockito.spy(QueueCommandManager.newInstance()); + Mockito.when(cmdManager.hostname()).thenReturn("host1"); + Mockito.when(cmdManager.user()).thenReturn("user"); + pipelineTask = Mockito.mock(PipelineTask.class); + Mockito.when(pipelineTask.taskBaseName()).thenReturn("100-200-tps"); + } + + @Test + public void testPopulateJobIds() { + + QstatParser qstatParser = new QstatParser(cmdManager); + // mock the returns for each of the 3 tasks: note that the first task will + // return exactly 2 jobs, the second task will return 3 jobs (different server + // name for one of them), and the 3rd job will return nothing. + + String qstat1a = "1234567.batch user low 100-200-tps.0 5 5 04:00 R 02:33 254%"; + String qstat1b = "1234568.batch user low 100-200-tps.1 5 5 04:00 R 02:33 254%"; + String qstat2a = "1234587.batch user low 100-201-tps.0 5 5 04:00 R 02:33 254%"; + String qstat2b = "1234597.batch user low 100-201-tps.1 5 5 04:00 R 02:33 254%"; + String qstat2c = "1234599.batch user low 100-201-tps 5 5 04:00 R 02:33 254%"; + + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-200-tps" }, + qstat1a, qstat1b); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-201-tps" }, + qstat2a, qstat2b, qstat2c); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-202-tps" }, + (String[]) null); + + // mock the return of the server names + String[] jobOrOwner = { "Job:", "Job_Owner" }; + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567 1234568", jobOrOwner, + "Job: 1234567.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234568.batch.example.com", " Job_Owner = user@host1.example.com"); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587 1234597 1234599", jobOrOwner, + "Job: 1234587.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234597.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234599.batch.example.com", " Job_Owner = user@host2.example.com"); + + // Create the RemoteJobInformation instances for a task. + RemoteJobInformation task200p0Information = new RemoteJobInformation("pbsLogfile", + "100-200-tps.0"); + RemoteJobInformation task200p1Information = new RemoteJobInformation("pbsLogfile", + "100-200-tps.1"); + qstatParser.populateJobIds(pipelineTask, + List.of(task200p0Information, task200p1Information)); + assertEquals(1234567L, task200p0Information.getJobId()); + assertEquals(1234568L, task200p1Information.getJobId()); + } + + @Test + public void testJobIdByName() { + + QstatParser qstatParser = new QstatParser(cmdManager); + // mock the returns for each of the 3 tasks: note that the first task will + // return exactly 2 jobs, the second task will return 3 jobs (different server + // name for one of them), and the 3rd job will return nothing. + + String qstat1a = "1234567.batch user low 100-200-tps.0 5 5 04:00 R 02:33 254%"; + String qstat1b = "1234568.batch user low 100-200-tps.1 5 5 04:00 R 02:33 254%"; + String qstat2a = "1234587.batch user low 100-201-tps.0 5 5 04:00 R 02:33 254%"; + String qstat2b = "1234597.batch user low 100-201-tps.1 5 5 04:00 R 02:33 254%"; + String qstat2c = "1234599.batch user low 100-201-tps 5 5 04:00 R 02:33 254%"; + + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-200-tps" }, + qstat1a, qstat1b); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-201-tps" }, + qstat2a, qstat2b, qstat2c); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-u user", new String[] { "100-202-tps" }, + (String[]) null); + + // mock the return of the server names + String[] jobOrOwner = { "Job:", "Job_Owner" }; + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234567 1234568", jobOrOwner, + "Job: 1234567.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234568.batch.example.com", " Job_Owner = user@host1.example.com"); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 1234587 1234597 1234599", jobOrOwner, + "Job: 1234587.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234597.batch.example.com", " Job_Owner = user@host1.example.com", + "Job: 1234599.batch.example.com", " Job_Owner = user@host2.example.com"); + + // Obtain the Map for task 100-200-tps + Map jobIdByName = qstatParser.jobIdByName("100-200-tps"); + assertTrue(jobIdByName.containsKey("100-200-tps.0")); + assertEquals(1234567L, jobIdByName.get("100-200-tps.0").longValue()); + assertTrue(jobIdByName.containsKey("100-200-tps.1")); + assertEquals(1234568L, jobIdByName.get("100-200-tps.1").longValue()); + assertEquals(2, jobIdByName.size()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerForUnitTests.java b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerForUnitTests.java index b630900..9b1b164 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerForUnitTests.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerForUnitTests.java @@ -1,10 +1,8 @@ package gov.nasa.ziggy.module.remote; -import java.util.HashMap; +import java.util.ArrayList; +import java.util.Collection; import java.util.List; -import java.util.Map; - -import gov.nasa.ziggy.module.StateFile; /** * Subclass of QueueCommandManager that can be used for unit tests. Its qstat() and qdel() methods @@ -15,7 +13,7 @@ */ public class QueueCommandManagerForUnitTests extends QueueCommandManager { - private Map queueDeleteCommands = new HashMap<>(); + private List queueDeleteCommands = new ArrayList<>(); @Override protected List qstat(String commandString, String... strings) { @@ -38,31 +36,36 @@ public String hostname() { } // Store information about a call for a task deletion, and return an appropriate return code. - // For task IDs between 101 and 200, return code is 1; all other cases, 0. + // When all job IDS are between 100 and 200, return code is 0; all other cases, 1. @Override - public int deleteJobsForStateFile(StateFile stateFile) { - long taskId = stateFile.getPipelineTaskId(); - int returnCode = taskId > 100 && taskId <= 200 ? 1 : 0; - queueDeleteCommands.put(taskId, new QueueDeleteCommand(stateFile, returnCode)); - return returnCode; + public int deleteJobsByJobId(Collection jobIds) { + super.deleteJobsByJobId(jobIds); + for (long jobId : jobIds) { + if (jobId < 100 || jobId > 200) { + queueDeleteCommands.add(new QueueDeleteCommand(jobIds, 1)); + return 1; + } + } + queueDeleteCommands.add(new QueueDeleteCommand(jobIds, 0)); + return 0; } - public Map getQueueDeleteCommands() { + public List getQueueDeleteCommands() { return queueDeleteCommands; } public static class QueueDeleteCommand { - private final StateFile stateFile; + private final Collection jobIds; private final int returnCode; - public QueueDeleteCommand(StateFile stateFile, int returnCode) { - this.stateFile = stateFile; + public QueueDeleteCommand(Collection jobIds, int returnCode) { + this.jobIds = jobIds; this.returnCode = returnCode; } - public StateFile getStateFile() { - return stateFile; + public Collection getJobIds() { + return jobIds; } public int getReturnCode() { diff --git a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java index af0d63f..8982fad 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java @@ -2,6 +2,7 @@ import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_QUEUE_COMMAND_CLASS; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -16,7 +17,6 @@ import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.RemoteJob; import gov.nasa.ziggy.pipeline.definition.RemoteJob.RemoteJobQstatInfo; @@ -91,36 +91,6 @@ public void testServerNames() { assertEquals("host1", jobIdServerMap.get(9101189L)); } - /** - * Tests getQstatInfoFromJobId method. - */ - @Test - public void testGetQstatInfoFromJobId() { - - // create the strings that get returned - String header1 = " Req'd Elap"; - String header2 = "JobID User Queue Jobname TSK Nds wallt S wallt Eff"; - String header3 = "-------------- ------- ----- ------------- --- --- ----- - ----- ----"; - String content1 = "9101154.batch user low tps-340-23787 5 5 04:00 F 01:34 419%"; - String content2 = "9101189.batch user low tps-340-23783 5 5 04:00 F 02:54 242%"; - - mockQstatCall("-x 9101189 9101154", null, header1, header2, header3, content1, content2); - - // create the list of job IDs -- note, create in a different order from the - // return order - List jobIds = new ArrayList<>(); - jobIds.add(9101189L); - jobIds.add(9101154L); - - // execute the method - Map jobNameToQstatOutputMap = cmdManager.getQstatInfoByJobNameMap(jobIds); - assertEquals(2, jobNameToQstatOutputMap.size()); - assertTrue(jobNameToQstatOutputMap.containsKey("tps-340-23787")); - assertTrue(jobNameToQstatOutputMap.get("tps-340-23787").equals(content1)); - assertTrue(jobNameToQstatOutputMap.containsKey("tps-340-23783")); - assertTrue(jobNameToQstatOutputMap.get("tps-340-23783").equals(content2)); - } - /** * Tests the method that returns an exit status for a job. */ @@ -134,67 +104,6 @@ public void testExitStatus() { assertEquals(0, cmdManager.exitStatus(9101154L).intValue()); } - /** - * Tests the method that returns an exit comment for a job - */ - @Test - public void testExitComment() { - - // create the string that gets returned - String returnString = " comment = test comment"; - mockQstatCall("-xf 9101154", new String[] { "comment" }, returnString); - - // execute the method - String exitComment = cmdManager.exitComment(9101154L); - assertTrue(exitComment.equals("test comment")); - } - - /** - * Test deleteJobsForPipelineTasks() method. - */ - @Test - public void testDeleteJobsForPipelineTasks() { - - PipelineTask task1 = Mockito.mock(PipelineTask.class); - PipelineTask task2 = Mockito.mock(PipelineTask.class); - PipelineTask task3 = Mockito.mock(PipelineTask.class); - - Mockito.when(task1.taskBaseName()).thenReturn(PipelineTask.taskBaseName(50L, 100L, "tps")); - Mockito.when(task2.taskBaseName()).thenReturn(PipelineTask.taskBaseName(50L, 101L, "tps")); - Mockito.when(task3.taskBaseName()).thenReturn(PipelineTask.taskBaseName(50L, 102L, "tps")); - - List tasks = new ArrayList<>(); - tasks.add(task1); - tasks.add(task2); - tasks.add(task3); - - // set up the returns for the qstat commands that are looking for the tasks in - // the queue -- NB, there is no job in the queue for task 3. - String jobName = task1.taskBaseName(); - String[] grepArgs = { jobName }; - mockQstatCall("-u user", grepArgs, qstatOutputLine(task1, 1234567L)); - jobName = task2.taskBaseName(); - grepArgs = new String[] { jobName }; - mockQstatCall("-u user", grepArgs, qstatOutputLine(task1, 7654321L)); - jobName = task3.taskBaseName(); - grepArgs = new String[] { jobName }; - mockQstatCall("-u user", grepArgs, (String[]) null); - - grepArgs = new String[] { "Job:", "Job_Owner" }; - mockQstatCall("-xf 1234567", grepArgs, "Job: 1234567.batch.example.com", - " Job_Owner = user@host.example.com"); - mockQstatCall("-xf 7654321", grepArgs, "Job: 7654321.batch.example.com", - " Job_Owner = user@host.example.com"); - - // execute the command - cmdManager.deleteJobsForPipelineTasks(tasks); - - // verify that the qdel command was called as expected -- note that the task1 - // and task2 job IDs are present, there is an extra space due to null task3 job - // ID, etc. - Mockito.verify(cmdManager, Mockito.times(1)).qdel("1234567 7654321 "); - } - /** * Test that an appropriate {@link RemoteJobQstatInfo} instance is returned when all necessary * information is present in the return from qstat. @@ -259,10 +168,20 @@ public void testDeleteJobsByJobId() { Mockito.verify(cmdManager, Mockito.times(1)).qdel("1234567 1234568 1234587 "); } - // generates the output line from qstat for a given task name and job ID - private String qstatOutputLine(PipelineTask task, long jobId) { - return String.format("%d.batch user low %s 5 5 04:00 F 02:33 254%%", jobId, - task.taskBaseName()); + @Test + public void testRemoteJobInformation() { + RemoteJob remoteJob = new RemoteJob(); + remoteJob.setFinished(false); + remoteJob.setJobId(1234567L); + mockQstatCall("-xf 1234567", + new String[] { QueueCommandManager.JOBNAME, QueueCommandManager.OUTPUT_PATH }, + " Job_Name = dv-118-36426.0", + " Output_Path = draco.nas.nasa.gov:/non/existent/path"); + RemoteJobInformation remoteJobInformation = cmdManager.remoteJobInformation(remoteJob); + assertNotNull(remoteJobInformation); + assertEquals(1234567L, remoteJobInformation.getJobId()); + assertEquals("/non/existent/path", remoteJobInformation.getLogFile()); + assertEquals("dv-118-36426.0", remoteJobInformation.getJobName()); } // Mocks the cmdManager in this test to return the correct list of strings when diff --git a/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java index 72d2502..82d4703 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java @@ -2,29 +2,37 @@ import static gov.nasa.ziggy.services.config.PropertyName.RESULTS_DIR; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; import java.io.File; import java.io.IOException; +import java.util.HashSet; +import java.util.Set; import java.util.concurrent.ExecutionException; import java.util.concurrent.Future; +import org.apache.commons.collections.CollectionUtils; import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.TaskConfiguration; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.database.DatabaseService; import gov.nasa.ziggy.services.database.SingleThreadExecutor; @@ -39,9 +47,14 @@ public class RemoteExecutorTest { private GenericRemoteExecutor executor; private PipelineTask pipelineTask; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; private TaskConfiguration taskConfigurationManager; private PipelineInstance pipelineInstance; + private QstatParser qstatParser; private static Future futureVoid; + private RemoteJob completeRemoteJob; + private RemoteJob incompleteRemoteJob; + private RemoteJobInformation incompleteRemoteJobInformation; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -67,9 +80,11 @@ public void setup() throws InterruptedException, ExecutionException { pipelineTask = mock(PipelineTask.class); pipelineInstance = mock(PipelineInstance.class); pipelineTaskOperations = mock(PipelineTaskOperations.class); + pipelineTaskDataOperations = mock(PipelineTaskDataOperations.class); taskConfigurationManager = mock(TaskConfiguration.class); executor = new GenericRemoteExecutor(pipelineTask); futureVoid = mock(Future.class); + qstatParser = mock(QstatParser.class); when(pipelineTask.getPipelineInstanceId()).thenReturn(10L); when(pipelineTask.getModuleName()).thenReturn("modulename"); @@ -80,11 +95,19 @@ public void setup() throws InterruptedException, ExecutionException { .thenReturn(new PipelineDefinitionNode()); when(pipelineTaskOperations.pipelineInstance(ArgumentMatchers.any(PipelineTask.class))) .thenReturn(pipelineInstance); + SubtaskCounts subtaskCounts = new SubtaskCounts(500, 400, 0); + when(pipelineTaskDataOperations.subtaskCounts(pipelineTask)).thenReturn(subtaskCounts); when(pipelineInstance.getId()).thenReturn(10L); when(taskConfigurationManager.getSubtaskCount()).thenReturn(500); - when(pipelineTask.getTotalSubtaskCount()).thenReturn(500); - when(pipelineTask.getCompletedSubtaskCount()).thenReturn(400); when(futureVoid.get()).thenReturn(null); + completeRemoteJob = new RemoteJob(); + completeRemoteJob.setFinished(true); + completeRemoteJob.setJobId(1234567L); + incompleteRemoteJob = new RemoteJob(); + incompleteRemoteJob.setFinished(false); + incompleteRemoteJob.setJobId(1234568L); + incompleteRemoteJobInformation = new RemoteJobInformation("test1", "test2"); + incompleteRemoteJobInformation.setJobId(1234568L); } @After @@ -105,21 +128,12 @@ public void testExecuteAlgorithmFirstIteration() { GenericRemoteExecutor gExecutor = executor; assertEquals(500, gExecutor.totalSubtaskCount); checkPbsParameterValues(remoteExecutionConfigurationForPipelineTask(), - gExecutor.pbsParameters); - assertEquals(RemoteNodeDescriptor.BROADWELL, gExecutor.pbsParameters.getArchitecture()); + gExecutor.getPbsParameters()); + assertEquals(RemoteNodeDescriptor.BROADWELL, + gExecutor.getPbsParameters().getArchitecture()); // The correct calls should have occurred. assertEquals(pipelineTask, gExecutor.pipelineTask()); - StateFile stateFile = gExecutor.stateFile(); - assertEquals(10L, stateFile.getPipelineInstanceId()); - assertEquals(50L, stateFile.getPipelineTaskId()); - assertEquals(500, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumFailed()); - assertEquals(0, stateFile.getNumComplete()); - - checkStateFileValues(gExecutor.pbsParameters, stateFile); - - assertEquals(stateFile, gExecutor.monitoredStateFile); } @Test @@ -136,23 +150,47 @@ public void testExecuteAlgorithmLaterIteration() { GenericRemoteExecutor gExecutor = executor; assertEquals(100, gExecutor.totalSubtaskCount); checkPbsParameterValues(remoteExecutionConfigurationFromDatabase(), - gExecutor.pbsParameters); - assertEquals(RemoteNodeDescriptor.ROME, gExecutor.pbsParameters.getArchitecture()); + gExecutor.getPbsParameters()); + assertEquals(RemoteNodeDescriptor.ROME, gExecutor.getPbsParameters().getArchitecture()); // The correct calls should have occurred, including that // the state file gets the full number of subtasks and no information about failed // or complete (that has to be generated at runtime by the remote job itself). assertEquals(pipelineTask, gExecutor.pipelineTask()); - StateFile stateFile = gExecutor.stateFile(); - assertEquals(10L, stateFile.getPipelineInstanceId()); - assertEquals(50L, stateFile.getPipelineTaskId()); - assertEquals(500, stateFile.getNumTotal()); - assertEquals(0, stateFile.getNumFailed()); - assertEquals(0, stateFile.getNumComplete()); + } + + @Test + public void testResumeMonitoringNoRemoteJobs() { + Mockito.when(pipelineTaskDataOperations.remoteJobs(pipelineTask)) + .thenReturn(new HashSet<>()); + RemoteExecutor remoteExecutor = new GenericRemoteExecutor(pipelineTask); + assertFalse(remoteExecutor.resumeMonitoring()); + assertTrue(CollectionUtils.isEmpty(remoteExecutor.getRemoteJobsInformation())); + } - checkStateFileValues(gExecutor.pbsParameters, stateFile); + @Test + public void testResumeMonitoringNoIncompleteRemoteJobs() { + Mockito.when(pipelineTaskDataOperations.remoteJobs(pipelineTask)) + .thenReturn(Set.of(completeRemoteJob)); + RemoteExecutor remoteExecutor = new GenericRemoteExecutor(pipelineTask); + assertFalse(remoteExecutor.resumeMonitoring()); + assertTrue(CollectionUtils.isEmpty(remoteExecutor.getRemoteJobsInformation())); + } - assertEquals(stateFile, gExecutor.monitoredStateFile); + @Test + public void testResumeMonitoring() { + Mockito.when(pipelineTaskDataOperations.remoteJobs(pipelineTask)) + .thenReturn(Set.of(completeRemoteJob, incompleteRemoteJob)); + Mockito.when(qstatParser.remoteJobInformation(incompleteRemoteJob)) + .thenReturn(incompleteRemoteJobInformation); + RemoteExecutor remoteExecutor = new GenericRemoteExecutor(pipelineTask); + assertTrue(remoteExecutor.resumeMonitoring()); + assertFalse(CollectionUtils.isEmpty(remoteExecutor.getRemoteJobsInformation())); + RemoteJobInformation remoteJobInformation = remoteExecutor.getRemoteJobsInformation() + .get(0); + assertEquals("test1", remoteJobInformation.getLogFile()); + assertEquals("test2", remoteJobInformation.getJobName()); + assertEquals(1234568L, remoteJobInformation.getJobId()); } private void checkPbsParameterValues( @@ -161,16 +199,7 @@ private void checkPbsParameterValues( assertEquals(executionResources.getQueueName(), pParameters.getQueueName()); assertEquals(executionResources.getMinCoresPerNode(), pParameters.getMinCoresPerNode()); assertEquals(executionResources.getMinGigsPerNode(), pParameters.getMinGigsPerNode(), 1e-3); - assertEquals(executionResources.getMaxNodes(), pParameters.getRequestedNodeCount()); - } - - private void checkStateFileValues(PbsParameters pParameters, StateFile stateFile) { - assertEquals(pParameters.getArchitecture().getNodeName(), - stateFile.getRemoteNodeArchitecture()); - assertEquals(pParameters.getQueueName(), stateFile.getQueueName()); - assertEquals(pParameters.getMinGigsPerNode(), stateFile.getMinGigsPerNode(), 1e-3); - assertEquals(pParameters.getMinCoresPerNode(), stateFile.getMinCoresPerNode()); - assertEquals(pParameters.getRequestedNodeCount(), stateFile.getRequestedNodeCount()); + assertEquals(2, pParameters.getRequestedNodeCount()); } // Parameters that come from the PipelineTask @@ -212,8 +241,6 @@ private PipelineDefinitionNodeExecutionResources remoteExecutionConfigurationFro private class GenericRemoteExecutor extends RemoteExecutor { int totalSubtaskCount; - PbsParameters pbsParameters; - StateFile monitoredStateFile; public GenericRemoteExecutor(PipelineTask pipelineTask) { super(pipelineTask); @@ -223,31 +250,37 @@ public PipelineTask pipelineTask() { return pipelineTask; } - public StateFile stateFile() { - return getStateFile(); - } - @Override public PbsParameters generatePbsParameters( PipelineDefinitionNodeExecutionResources remoteParameters, int totalSubtaskCount) { this.totalSubtaskCount = totalSubtaskCount; - pbsParameters = remoteParameters.pbsParametersInstance(); + PbsParameters pbsParameters = remoteParameters.pbsParametersInstance(); + pbsParameters.populateResourceParameters(remoteParameters, totalSubtaskCount); return pbsParameters; } @Override - public void addToMonitor(StateFile stateFile) { - monitoredStateFile = stateFile; + public void addToMonitor() { } @Override - protected void submitForExecution(StateFile stateFile) { - addToMonitor(stateFile); + protected void submitForExecution() { + addToMonitor(); } @Override protected PipelineTaskOperations pipelineTaskOperations() { return pipelineTaskOperations; } + + @Override + protected PipelineTaskDataOperations pipelineTaskDataOperations() { + return pipelineTaskDataOperations; + } + + @Override + protected QstatParser qstatParser() { + return qstatParser; + } } } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java b/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java index eaf8609..0450b09 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java @@ -11,11 +11,11 @@ import java.util.ArrayList; import java.util.List; +import java.util.Set; import org.junit.Before; import org.junit.Test; import org.mockito.ArgumentMatchers; -import org.mockito.Mockito; import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.module.DatastoreDirectoryPipelineInputs; @@ -55,8 +55,8 @@ public class PipelineTaskInformationTest { private ParameterSet instanceParSet2 = new ParameterSet(instancePars2Name); private String moduleParsName = "Module Pars"; private ParameterSet moduleParSet = new ParameterSet(moduleParsName); - private PipelineTask p1 = Mockito.spy(PipelineTask.class); - private PipelineTask p2 = Mockito.spy(PipelineTask.class); + private PipelineTask p1 = spy(PipelineTask.class); + private PipelineTask p2 = spy(PipelineTask.class); private SubtaskInformation s1, s2; private PipelineDefinitionNode node; private PipelineDefinition pipelineDefinition; @@ -106,7 +106,9 @@ public void setup() { // Set up unit of work generation doReturn(uowGenerator).when(pipelineTaskInformation).unitOfWorkGenerator(node); UnitOfWork u1 = new UnitOfWork(); + u1.setParameters(Set.of(new Parameter("param1", "value1"))); UnitOfWork u2 = new UnitOfWork(); + u2.setParameters(Set.of(new Parameter("param2", "value2"))); List uowList = new ArrayList<>(); uowList.add(u1); uowList.add(u2); @@ -116,8 +118,8 @@ ArgumentMatchers. any(), ArgumentMatchers. any()); // Set up pipeline task generation - Mockito.doReturn(1L).when(p1).getId(); - Mockito.doReturn(2L).when(p2).getId(); + doReturn(1L).when(p1).getId(); + doReturn(2L).when(p2).getId(); doReturn(p1).doReturn(p2) .when(pipelineTaskInformation) .pipelineTask(any(PipelineInstance.class), any(PipelineInstanceNode.class), diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/ExecutionClockTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/ExecutionClockTest.java new file mode 100644 index 0000000..254b781 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/ExecutionClockTest.java @@ -0,0 +1,117 @@ +package gov.nasa.ziggy.pipeline.definition; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; + +import java.text.ParseException; + +import org.junit.Test; + +import gov.nasa.ziggy.util.SystemProxy; + +/** + * Performs unit tests for {@link ExecutionClock}. + * + * @author PT + * @author Bill Wohler + */ +public class ExecutionClockTest { + + private static final long MILLIS_PER_SECOND = 1000L; + private static final int SECONDS_PER_MINUTE = 60; + private static final int MINUTES_PER_HOUR = 60; + + @Test + public void testStartExecutionClockOnly() { + ExecutionClock executionClock = new ExecutionClock(); + assertEquals("-", executionClock.toString()); + + // 00:00:01 + SystemProxy.setUserTime(1 * MILLIS_PER_SECOND); + executionClock.start(); + assertTrue(executionClock.isRunning()); + + // 01:01:01 + SystemProxy + .setUserTime((1 * MINUTES_PER_HOUR * SECONDS_PER_MINUTE + 1 * SECONDS_PER_MINUTE + 1) + * MILLIS_PER_SECOND); + assertEquals("01:01:00", executionClock.toString()); + } + + @Test + public void testStopExecutionClock() { + ExecutionClock executionClock = new ExecutionClock(); + + // 00:00:01 + SystemProxy.setUserTime(1 * MILLIS_PER_SECOND); + executionClock.start(); + assertTrue(executionClock.isRunning()); + + // 00:01:00 + SystemProxy.setUserTime(1 * SECONDS_PER_MINUTE * MILLIS_PER_SECOND); + executionClock.stop(); + assertFalse(executionClock.isRunning()); + assertEquals("00:00:59", executionClock.toString()); + } + + @Test + public void testStopExecutionClockTwice() throws ParseException { + ExecutionClock executionClock = new ExecutionClock(); + + // 00:00:01 + SystemProxy.setUserTime(1 * MILLIS_PER_SECOND); + executionClock.start(); + assertTrue(executionClock.isRunning()); + + // 00:00:01 + SystemProxy.setUserTime(1 * MILLIS_PER_SECOND); + assertEquals("00:00:00", executionClock.toString()); + + // 00:01:00 + SystemProxy.setUserTime(1 * SECONDS_PER_MINUTE * MILLIS_PER_SECOND); + executionClock.stop(); + assertFalse(executionClock.isRunning()); + assertEquals("00:00:59", executionClock.toString()); + + // 00:01:30 + SystemProxy.setUserTime((1 * SECONDS_PER_MINUTE + 30) * MILLIS_PER_SECOND); + executionClock.start(); + assertTrue(executionClock.isRunning()); + + // 00:01:45 + SystemProxy.setUserTime((1 * SECONDS_PER_MINUTE + 45) * MILLIS_PER_SECOND); + assertEquals("00:01:14", executionClock.toString()); + + // 00:02:30 + SystemProxy.setUserTime((2 * SECONDS_PER_MINUTE + 30) * MILLIS_PER_SECOND); + executionClock.stop(); + assertFalse(executionClock.isRunning()); + assertEquals("00:01:59", executionClock.toString()); + } + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + ExecutionClock executionClock1 = new ExecutionClock(); + ExecutionClock executionClock2 = new ExecutionClock(); + assertEquals(executionClock1.hashCode(), executionClock2.hashCode()); + assertTrue(executionClock1.equals(executionClock1)); + assertTrue(executionClock1.equals(executionClock2)); + + executionClock2.start(); + assertNotEquals(executionClock1.hashCode(), executionClock2.hashCode()); + assertFalse(executionClock1.equals(executionClock2)); + + executionClock2.stop(); + executionClock2.stop(); + executionClock2.start(); + executionClock2.start(); + assertNotEquals(executionClock1.hashCode(), executionClock2.hashCode()); + assertFalse(executionClock1.equals(executionClock2)); + + assertFalse(executionClock1.equals(null)); + assertFalse(executionClock1.equals("a string")); + } +} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporterTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporterTest.java index d92ac0f..31fc62c 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporterTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionImporterTest.java @@ -1,6 +1,7 @@ package gov.nasa.ziggy.pipeline.definition; import static gov.nasa.ziggy.ZiggyUnitTestUtils.TEST_DATA; +import static gov.nasa.ziggy.services.config.PropertyName.PIPELINE_HOME_DIR; import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -16,7 +17,7 @@ import java.util.Set; import java.util.stream.Collectors; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -62,6 +63,10 @@ public class PipelineDefinitionImporterTest { public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, DirectoryProperties.ziggyCodeBuildDir().toString()); + @Rule + public ZiggyPropertyRule pipelineHomeDirPropertyRule = new ZiggyPropertyRule(PIPELINE_HOME_DIR, + DirectoryProperties.ziggyCodeBuildDir().toString()); + @Before public void setUp() { pipelineDefinitionFile = TEST_DATA.resolve("configuration") diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java index b291a0f..01574a9 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java @@ -83,7 +83,8 @@ public void testMarshaller() throws JAXBException, IOException { marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE); marshaller.marshal(node, xmlFile); assertTrue(xmlFile.exists()); - List xmlContent = Files.readAllLines(xmlFile.toPath(), ZiggyFileUtils.ZIGGY_CHARSET); + List xmlContent = Files.readAllLines(xmlFile.toPath(), + ZiggyFileUtils.ZIGGY_CHARSET); assertEquals(9, xmlContent.size()); List nodeContent = nodeContent(xmlContent, " xmlContent = Files.readAllLines(xmlFile.toPath(), ZiggyFileUtils.ZIGGY_CHARSET); + List xmlContent = Files.readAllLines(xmlFile.toPath(), + ZiggyFileUtils.ZIGGY_CHARSET); List pipelineContents = pipelineContent(xmlContent, " 10000000L); - assertEquals(0L, pipelineExecutionTime.getPriorProcessingExecutionTimeMillis()); - assertEquals(0L, pipelineExecutionTime.getEndProcessingTime().getTime()); - } - - @Test - public void testStopExecutionClockSetsEndProcessingTime() { - - // When the execution clock is stopped, it should set the end processing time to the - // current time at that instant. - PipelineExecutionTime pipelineExecutionTime = new PipelineExecutionTimeImplForTesting(); - Date currentDate = new Date(); - pipelineExecutionTime.stopExecutionClock(); - assertTrue( - pipelineExecutionTime.getEndProcessingTime().getTime() - currentDate.getTime() < 10L); - } - - @Test - public void testStopExecutionClock() throws ParseException { - - // When the stop execution clock is called the first time, the time spent in the first - // processing attempt is moved into the prior processing time and the start time for - // the current event is set to -1. - PipelineExecutionTime pipelineExecutionTime = Mockito - .spy(PipelineExecutionTimeImplForTesting.class); - Date startDate = new SimpleDateFormat("yyMMddHHmmss").parse("20211229000000"); - pipelineExecutionTime.setStartProcessingTime(startDate); - pipelineExecutionTime.setCurrentExecutionStartTimeMillis( - pipelineExecutionTime.getStartProcessingTime().getTime()); - Mockito.when(pipelineExecutionTime.getEndProcessingTime()) - .thenReturn(new SimpleDateFormat("yyMMddHHmmss").parse("20211229010000")); - assertEquals(0L, pipelineExecutionTime.getPriorProcessingExecutionTimeMillis()); - pipelineExecutionTime.stopExecutionClock(); - assertEquals(-1L, pipelineExecutionTime.getCurrentExecutionStartTimeMillis()); - assertEquals(60000L, pipelineExecutionTime.getPriorProcessingExecutionTimeMillis()); - - // When a second processing attempt is performed,the stopExecutionClock() method - // increments the prior processing time by the amount of the most recent processing - // duration. - pipelineExecutionTime.setCurrentExecutionStartTimeMillis(startDate.getTime()); - pipelineExecutionTime.stopExecutionClock(); - assertEquals(120000L, pipelineExecutionTime.getPriorProcessingExecutionTimeMillis()); - assertEquals(-1L, pipelineExecutionTime.getCurrentExecutionStartTimeMillis()); - } - - @Test - public void testTotalTimeAllAttemptsMillis() throws ParseException { - - // When the total time from all processing attempts is requested and we're on the - // first processing attempt, the total is just the time from the start of current - // attempt to the current instant. - PipelineExecutionTime pipelineExecutionTime = Mockito - .spy(PipelineExecutionTimeImplForTesting.class); - Date startDate = new SimpleDateFormat("yyMMddHHmmss").parse("20211229000000"); - pipelineExecutionTime.setStartProcessingTime(startDate); - pipelineExecutionTime.setCurrentExecutionStartTimeMillis( - pipelineExecutionTime.getStartProcessingTime().getTime()); - Mockito.when(pipelineExecutionTime.currentTimeMillis()) - .thenReturn(new SimpleDateFormat("yyMMddHHmmss").parse("20211229010000").getTime()); - long exeTime = pipelineExecutionTime.totalExecutionTimeAllAttemptsMillis(); - assertEquals(60000L, exeTime); - - // On subsequent attempts, the value from prior processing attempts is added to the - // current processing attempt. - pipelineExecutionTime.setPriorProcessingExecutionTimeMillis(100L); - exeTime = pipelineExecutionTime.totalExecutionTimeAllAttemptsMillis(); - assertEquals(60100L, exeTime); - } - - public static class PipelineExecutionTimeImplForTesting implements PipelineExecutionTime { - - private Date startProcessingTime = new Date(0); - private Date endProcessingTime = new Date(0); - private long totalExecutionTimeMillis; - private long currentExecutionStartTimeMillis; - - public PipelineExecutionTimeImplForTesting() { - currentExecutionStartTimeMillis = -1; - } - - @Override - public void setStartProcessingTime(Date date) { - startProcessingTime = date; - } - - @Override - public Date getStartProcessingTime() { - return startProcessingTime; - } - - @Override - public void setEndProcessingTime(Date date) { - endProcessingTime = date; - } - - @Override - public Date getEndProcessingTime() { - return endProcessingTime; - } - - @Override - public void setPriorProcessingExecutionTimeMillis(long executionTimeMillis) { - totalExecutionTimeMillis = executionTimeMillis; - } - - @Override - public long getPriorProcessingExecutionTimeMillis() { - return totalExecutionTimeMillis; - } - - @Override - public void setCurrentExecutionStartTimeMillis(long linuxStartTimeCurrentExecution) { - currentExecutionStartTimeMillis = linuxStartTimeCurrentExecution; - } - - @Override - public long getCurrentExecutionStartTimeMillis() { - return currentExecutionStartTimeMillis; - } - } -} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceTest.java new file mode 100644 index 0000000..8996676 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceTest.java @@ -0,0 +1,67 @@ +package gov.nasa.ziggy.pipeline.definition; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; + +import org.junit.Test; + +import gov.nasa.ziggy.pipeline.definition.PipelineInstance.State; + +/** + * Performs unit tests for {@link PipelineInstance}. + * + * @author Bill Wohler + */ +public class PipelineInstanceTest { + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + PipelineInstance pipelineInstance1 = new PipelineInstance(); + PipelineInstance pipelineInstance2 = new PipelineInstance(); + assertEquals(pipelineInstance1.hashCode(), pipelineInstance2.hashCode()); + assertTrue(pipelineInstance1.equals(pipelineInstance1)); + assertTrue(pipelineInstance1.equals(pipelineInstance2)); + + pipelineInstance2.setState(State.PROCESSING); + assertEquals(pipelineInstance1.hashCode(), pipelineInstance2.hashCode()); + assertTrue(pipelineInstance1.equals(pipelineInstance2)); + + pipelineInstance2.getExecutionClock().start(); + assertEquals(pipelineInstance1.hashCode(), pipelineInstance2.hashCode()); + assertTrue(pipelineInstance1.equals(pipelineInstance2)); + + pipelineInstance2.setId(42L); + assertNotEquals(pipelineInstance1.hashCode(), pipelineInstance2.hashCode()); + assertFalse(pipelineInstance1.equals(pipelineInstance2)); + + assertFalse(pipelineInstance1.equals(null)); + assertFalse(pipelineInstance1.equals("a string")); + } + + @Test + public void testTotalHashCodeEquals() { + PipelineInstance pipelineInstance1 = new PipelineInstance(); + PipelineInstance pipelineInstance2 = new PipelineInstance(); + assertEquals(pipelineInstance1.totalHashCode(), pipelineInstance2.totalHashCode()); + assertTrue(pipelineInstance1.totalEquals(pipelineInstance1)); + assertTrue(pipelineInstance1.totalEquals(pipelineInstance2)); + + pipelineInstance2.setState(State.PROCESSING); + assertNotEquals(pipelineInstance1.totalHashCode(), pipelineInstance2.totalHashCode()); + assertFalse(pipelineInstance1.totalEquals(pipelineInstance2)); + + pipelineInstance2.getExecutionClock().start(); + assertNotEquals(pipelineInstance1.totalHashCode(), pipelineInstance2.totalHashCode()); + assertFalse(pipelineInstance1.totalEquals(pipelineInstance2)); + + pipelineInstance2.setId(42L); + assertNotEquals(pipelineInstance1.totalHashCode(), pipelineInstance2.totalHashCode()); + assertFalse(pipelineInstance1.totalEquals(pipelineInstance2)); + + assertFalse(pipelineInstance1.totalEquals(null)); + assertFalse(pipelineInstance1.totalEquals("a string")); + } +} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java index da29d58..d0f4b74 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java @@ -93,7 +93,8 @@ public void testMarshaling() throws JAXBException, IOException { marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE); marshaller.marshal(module1, xmlFile); assertTrue(xmlFile.exists()); - List xmlContent = Files.readAllLines(xmlFile.toPath(), ZiggyFileUtils.ZIGGY_CHARSET); + List xmlContent = Files.readAllLines(xmlFile.toPath(), + ZiggyFileUtils.ZIGGY_CHARSET); assertContains(xmlContent, module1XmlString); marshaller.marshal(module2, xmlFile); diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDataTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDataTest.java new file mode 100644 index 0000000..38c5df7 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineTaskDataTest.java @@ -0,0 +1,285 @@ +package gov.nasa.ziggy.pipeline.definition; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; + +import java.util.Date; +import java.util.List; +import java.util.Set; + +import org.junit.Before; +import org.junit.Test; + +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; +import gov.nasa.ziggy.uow.UnitOfWork; + +/** + * Performs unit tests for {@link PipelineTaskData} and {@link PipelineTaskDisplayData}. + * + * @author Bill Wohler + */ + +public class PipelineTaskDataTest { + + private static final long PIPELINE_TASK_ID = 42L; + private static final long PIPELINE_INSTANCE_ID = 43L; + private static final long NOW = 1000L; + private static final Date CREATED = new Date(NOW); + private static final String MODULE_NAME = "moduleName"; + private static final String BRIEF_STATE = "briefState"; + private static final String ZIGGY_SOFTWARE_REVISION = "ziggyRevision"; + private static final String PIPELINE_SOFTWARE_REVISION = "pipelineRevision"; + private static final ProcessingStep PROCESSING_STEP = ProcessingStep.EXECUTING; + private static final String WORKER_HOST = "workerHost"; + private static final int WORKER_THREAD = 24; + private static final String WORKER_NAME = WORKER_HOST + ":" + WORKER_THREAD; + private static final ExecutionClock EXECUTION_CLOCK = new ExecutionClock(); + private static final int COMPLETED_SUBTASK_COUNT = 2; + private static final int FAILED_SUBTASK_COUNT = 3; + private static final int RUNNING_SUBTASK_COUNT = 1; + private static final int TOTAL_SUBTASK_COUNT = COMPLETED_SUBTASK_COUNT + FAILED_SUBTASK_COUNT + - RUNNING_SUBTASK_COUNT; + private static final int FAILURE_COUNT = 4; + private PipelineTask pipelineTask; + private PipelineTaskData pipelineTaskData; + private PipelineTaskDisplayData pipelineTaskDisplayData; + private Set remoteJobs; + private List pipelineTaskMetrics; + + @Before + public void setUp() { + pipelineTask = spy(PipelineTask.class); + doReturn(PIPELINE_TASK_ID).when(pipelineTask).getId(); + doReturn(PIPELINE_INSTANCE_ID).when(pipelineTask).getPipelineInstanceId(); + doReturn(MODULE_NAME).when(pipelineTask).getModuleName(); + doReturn(new UnitOfWork(BRIEF_STATE)).when(pipelineTask).getUnitOfWork(); + doReturn(CREATED).when(pipelineTask).getCreated(); + + pipelineTaskData = new PipelineTaskData(pipelineTask); + pipelineTaskData.setZiggySoftwareRevision(ZIGGY_SOFTWARE_REVISION); + pipelineTaskData.setPipelineSoftwareRevision(PIPELINE_SOFTWARE_REVISION); + pipelineTaskData.setProcessingStep(PROCESSING_STEP); + pipelineTaskData.setWorkerHost(WORKER_HOST); + pipelineTaskData.setWorkerThread(WORKER_THREAD); + pipelineTaskData.setCompletedSubtaskCount(COMPLETED_SUBTASK_COUNT); + pipelineTaskData.setFailedSubtaskCount(FAILED_SUBTASK_COUNT); + pipelineTaskData.setTotalSubtaskCount(TOTAL_SUBTASK_COUNT); + pipelineTaskData.setFailureCount(FAILURE_COUNT); + pipelineTaskMetrics = List + .of(new PipelineTaskMetric(MODULE_NAME, RUNNING_SUBTASK_COUNT, Units.RATE)); + pipelineTaskData.setPipelineTaskMetrics(pipelineTaskMetrics); + RemoteJob remoteJob = new RemoteJob(1); + remoteJob.setCostEstimate(42.0); + remoteJobs = Set.of(remoteJob); + pipelineTaskData.setRemoteJobs(remoteJobs); + + pipelineTaskDisplayData = new PipelineTaskDisplayData(pipelineTaskData); + } + + @Test + public void testCostEstimate() { + assertEquals(42.0, pipelineTaskDisplayData.costEstimate(), 0.0); + } + + @Test + public void testGetPipelineTask() { + assertEquals(pipelineTask, pipelineTaskDisplayData.getPipelineTask()); + } + + @Test + public void testGetPipelineTaskId() { + assertEquals(PIPELINE_TASK_ID, pipelineTaskDisplayData.getPipelineTaskId()); + } + + @Test + public void testGetPipelineInstanceId() { + assertEquals(PIPELINE_INSTANCE_ID, pipelineTaskDisplayData.getPipelineInstanceId()); + } + + @Test + public void testGetCreated() { + assertEquals(CREATED, pipelineTaskDisplayData.getCreated()); + } + + @Test + public void testGetModuleName() { + assertEquals(MODULE_NAME, pipelineTaskDisplayData.getModuleName()); + } + + @Test + public void testGetBriefState() { + assertEquals(BRIEF_STATE, pipelineTaskDisplayData.getBriefState()); + } + + @Test + public void testGetZiggySoftwareRevision() { + assertEquals(ZIGGY_SOFTWARE_REVISION, pipelineTaskDisplayData.getZiggySoftwareRevision()); + } + + @Test + public void testGetPipelineSoftwareRevision() { + assertEquals(PIPELINE_SOFTWARE_REVISION, + pipelineTaskDisplayData.getPipelineSoftwareRevision()); + } + + @Test + public void testGetWorkerName() { + assertEquals(WORKER_HOST, pipelineTaskData.getWorkerHost()); + assertEquals(WORKER_THREAD, pipelineTaskData.getWorkerThread()); + assertEquals(WORKER_NAME, pipelineTaskDisplayData.getWorkerName()); + } + + @Test + public void testGetExecutionClock() { + assertEquals(EXECUTION_CLOCK, pipelineTaskDisplayData.getExecutionClock()); + } + + @Test + public void testGetProcessingStep() { + assertEquals(PROCESSING_STEP, pipelineTaskDisplayData.getProcessingStep()); + } + + @Test + public void testGetDisplayProcessingStep() { + assertEquals(PROCESSING_STEP.toString(), + pipelineTaskDisplayData.getDisplayProcessingStep()); + + pipelineTaskData.setError(true); + pipelineTaskDisplayData = new PipelineTaskDisplayData(pipelineTaskData); + + assertEquals("ERROR - " + PROCESSING_STEP.toString(), + pipelineTaskDisplayData.getDisplayProcessingStep()); + } + + @Test + public void testIsError() { + assertEquals(false, pipelineTaskDisplayData.isError()); + + pipelineTaskData.setHaltRequested(true); + assertEquals(true, pipelineTaskData.isHaltRequested()); + + pipelineTaskData.setRetry(true); + assertEquals(true, pipelineTaskData.isRetry()); + } + + @Test + public void testGetTotalSubtaskCount() { + assertEquals(TOTAL_SUBTASK_COUNT, pipelineTaskDisplayData.getTotalSubtaskCount()); + } + + @Test + public void testGetCompletedSubtaskCount() { + assertEquals(COMPLETED_SUBTASK_COUNT, pipelineTaskDisplayData.getCompletedSubtaskCount()); + } + + @Test + public void testGetFailedSubtaskCount() { + assertEquals(FAILED_SUBTASK_COUNT, pipelineTaskDisplayData.getFailedSubtaskCount()); + } + + @Test + public void testGetFailureCount() { + assertEquals(FAILURE_COUNT, pipelineTaskDisplayData.getFailureCount()); + + pipelineTaskData.incrementFailureCount(); + pipelineTaskDisplayData = new PipelineTaskDisplayData(pipelineTaskData); + + assertEquals(FAILURE_COUNT + 1, pipelineTaskDisplayData.getFailureCount()); + } + + @Test + public void testAutoResubmitCount() { + pipelineTaskData.setAutoResubmitCount(42); + assertEquals(42, pipelineTaskData.getAutoResubmitCount()); + pipelineTaskData.incrementAutoResubmitCount(); + assertEquals(43, pipelineTaskData.getAutoResubmitCount()); + pipelineTaskData.resetAutoResubmitCount(); + assertEquals(0, pipelineTaskData.getAutoResubmitCount()); + } + + @Test + public void testAlgorithmType() { + pipelineTaskData.setAlgorithmType(AlgorithmType.REMOTE); + assertEquals(AlgorithmType.REMOTE, pipelineTaskData.getAlgorithmType()); + } + + @Test + public void testTaskLogIndex() { + pipelineTaskData.setTaskLogIndex(42); + assertEquals(42, pipelineTaskData.getTaskLogIndex()); + pipelineTaskData.incrementTaskLogIndex(); + assertEquals(43, pipelineTaskData.getTaskLogIndex()); + } + + @Test + public void testGetPipelineTaskMetrics() { + assertEquals(pipelineTaskMetrics, pipelineTaskDisplayData.getPipelineTaskMetrics()); + } + + @Test + public void testTaskExecutionLogs() { + + List taskExecutionLogs = List + .of(new TaskExecutionLog(WORKER_HOST, WORKER_THREAD)); + pipelineTaskData.setTaskExecutionLogs(taskExecutionLogs); + assertEquals(taskExecutionLogs, pipelineTaskData.getTaskExecutionLogs()); + } + + @Test + public void testGetRemoteJobs() { + assertEquals(remoteJobs, pipelineTaskDisplayData.getRemoteJobs()); + } + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + PipelineTaskDisplayData pipelineTaskDisplayData2 = new PipelineTaskDisplayData( + pipelineTaskData); + + assertEquals(pipelineTaskDisplayData.hashCode(), pipelineTaskDisplayData2.hashCode()); + assertTrue(pipelineTaskDisplayData.equals(pipelineTaskDisplayData)); + assertTrue(pipelineTaskDisplayData.equals(pipelineTaskDisplayData2)); + + PipelineTaskData pipelineTaskData2 = new PipelineTaskData(pipelineTask); + + assertEquals(pipelineTaskData.hashCode(), pipelineTaskData2.hashCode()); + assertTrue(pipelineTaskData.equals(pipelineTaskData)); + assertTrue(pipelineTaskData.equals(pipelineTaskData2)); + + PipelineTask pipelineTask2 = spy(PipelineTask.class); + doReturn(PIPELINE_TASK_ID + 1).when(pipelineTask2).getId(); + doReturn(new UnitOfWork(BRIEF_STATE + "foo")).when(pipelineTask2).getUnitOfWork(); + + pipelineTaskData2 = new PipelineTaskData(pipelineTask2); + pipelineTaskDisplayData2 = new PipelineTaskDisplayData(pipelineTaskData2); + assertNotEquals(pipelineTaskDisplayData.hashCode(), pipelineTaskDisplayData2.hashCode()); + assertFalse(pipelineTaskDisplayData.equals(pipelineTaskDisplayData2)); + + assertNotEquals(pipelineTaskData.hashCode(), pipelineTaskData2.hashCode()); + assertFalse(pipelineTaskData.equals(pipelineTaskData2)); + + assertFalse(pipelineTaskDisplayData.equals(null)); + assertFalse(pipelineTaskDisplayData.equals("a string")); + + assertFalse(pipelineTaskData.equals(null)); + assertFalse(pipelineTaskData.equals("a string")); + } + + @Test + public void testToFullString() { + assertEquals( + "PipelineTaskDisplayData: pipelineTaskId=" + PIPELINE_TASK_ID + ", moduleName=" + + MODULE_NAME + ", briefState=" + BRIEF_STATE, + pipelineTaskDisplayData.toFullString()); + } + + @Test + public void testToString() { + assertEquals(Long.toString(PIPELINE_TASK_ID), pipelineTaskDisplayData.toString()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/TaskCountsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/TaskCountsTest.java index 3ededff..560e7a8 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/TaskCountsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/TaskCountsTest.java @@ -1,15 +1,20 @@ package gov.nasa.ziggy.pipeline.definition; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertTrue; import java.util.ArrayList; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.junit.Before; import org.junit.Test; import gov.nasa.ziggy.pipeline.definition.TaskCounts.Counts; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineOperationsTestUtils; public class TaskCountsTest { @@ -19,7 +24,43 @@ public class TaskCountsTest { @Before public void setUp() { pipelineOperationsTestUtils = new PipelineOperationsTestUtils(); - pipelineOperationsTestUtils.setUpFivePipelineTasks(); + pipelineOperationsTestUtils.setUpFivePipelineTaskDisplayData(); + } + + @Test + public void testSubtaskCountsLabel() { + assertEquals("1/2", TaskCounts.subtaskCountsLabel(1, 2, 0)); + assertEquals("1/2 (1)", TaskCounts.subtaskCountsLabel(1, 2, 1)); + } + + @Test + public void testIsPipelineTasksComplete() { + TaskCounts taskCounts = new TaskCounts(); + assertEquals(true, taskCounts.isPipelineTasksComplete()); + taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()); + assertEquals(false, taskCounts.isPipelineTasksComplete()); + taskCounts = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(4))); + assertEquals(true, taskCounts.isPipelineTasksComplete()); + } + + @Test + public void testIsPipelineTasksExecutionComplete() { + TaskCounts taskCounts = new TaskCounts(); + assertEquals(false, taskCounts.isPipelineTasksExecutionComplete()); + taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()); + assertEquals(false, taskCounts.isPipelineTasksExecutionComplete()); + taskCounts = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(4))); + assertEquals(true, taskCounts.isPipelineTasksExecutionComplete()); + } + + @Test + public void testGetTaskCount() { + TaskCounts taskCounts = new TaskCounts(); + assertEquals(0, taskCounts.getTaskCount()); + taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()); + assertEquals(5, taskCounts.getTaskCount()); } @Test @@ -27,15 +68,15 @@ public void testGetModuleCounts() { TaskCounts taskCounts = new TaskCounts(); assertEquals(new HashMap<>(), taskCounts.getModuleCounts()); - taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTasks()); + taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()); Map moduleCounts = taskCounts.getModuleCounts(); assertEquals(5, moduleCounts.size()); - PipelineOperationsTestUtils.testCounts(1, 0, 0, 0, 10, 9, 1, moduleCounts.get("module1")); - PipelineOperationsTestUtils.testCounts(1, 0, 0, 0, 20, 18, 2, moduleCounts.get("module2")); - PipelineOperationsTestUtils.testCounts(0, 1, 0, 0, 30, 27, 3, moduleCounts.get("module3")); - PipelineOperationsTestUtils.testCounts(0, 0, 0, 1, 40, 36, 4, moduleCounts.get("module4")); - PipelineOperationsTestUtils.testCounts(0, 0, 1, 0, 50, 45, 5, moduleCounts.get("module5")); + testCounts(1, 0, 0, 0, 10, 9, 1, moduleCounts.get("module1")); + testCounts(1, 0, 0, 0, 20, 18, 2, moduleCounts.get("module2")); + testCounts(0, 1, 0, 0, 30, 27, 3, moduleCounts.get("module3")); + testCounts(0, 0, 0, 1, 40, 36, 4, moduleCounts.get("module4")); + testCounts(0, 0, 1, 0, 50, 45, 5, moduleCounts.get("module5")); } @Test @@ -43,7 +84,7 @@ public void testGetModuleNames() { TaskCounts taskCounts = new TaskCounts(); assertEquals(new ArrayList<>(), taskCounts.getModuleNames()); - taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTasks()); + taskCounts = new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()); assertEquals(5, taskCounts.getModuleNames().size()); assertEquals("module1", taskCounts.getModuleNames().get(0)); assertEquals("module2", taskCounts.getModuleNames().get(1)); @@ -53,38 +94,91 @@ public void testGetModuleNames() { } @Test - public void testGetTotalProcessingCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getProcessingTaskCount()); + public void testGetTotalCounts() { + TaskCounts taskCounts = new TaskCounts( + pipelineOperationsTestUtils.getPipelineTaskDisplayData()); + testCounts(2, 1, 1, 1, 150, 9 + 18 + 27 + 36 + 45, 15, taskCounts.getTotalCounts()); } + @SuppressWarnings("unlikely-arg-type") @Test - public void testGetTotalErrorCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getFailedTaskCount()); + public void testHashCodeEquals() { + TaskCounts taskCounts1 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(0))); + TaskCounts taskCounts2 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(0))); + + assertEquals(taskCounts1.hashCode(), taskCounts2.hashCode()); + assertTrue(taskCounts1.equals(taskCounts1)); + assertTrue(taskCounts1.equals(taskCounts2)); + + taskCounts2 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(1))); + assertNotEquals(taskCounts1.hashCode(), taskCounts2.hashCode()); + assertFalse(taskCounts1.equals(taskCounts2)); + + assertFalse(taskCounts1.equals(null)); + assertFalse(taskCounts1.equals("a string")); + + taskCounts1 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(2))); + taskCounts2 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(2))); + Counts totalCounts1 = taskCounts1.getTotalCounts(); + Counts totalCounts2 = taskCounts2.getTotalCounts(); + + assertEquals(totalCounts1.hashCode(), totalCounts2.hashCode()); + assertTrue(totalCounts1.equals(totalCounts1)); + assertTrue(totalCounts1.equals(totalCounts2)); + + taskCounts2 = new TaskCounts( + List.of(pipelineOperationsTestUtils.getPipelineTaskDisplayData().get(4))); + totalCounts2 = taskCounts2.getTotalCounts(); + assertNotEquals(totalCounts1.hashCode(), totalCounts2.hashCode()); + assertFalse(totalCounts1.equals(totalCounts2)); + + assertFalse(totalCounts1.equals(null)); + assertFalse(totalCounts1.equals("a string")); } @Test - public void testGetTotalCompletedCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getCompletedTaskCount()); + public void testToString() { + assertEquals( + "taskCount=5, waitingToRunTaskCount=2, completedTaskCount=1, failedTaskCount=1", + new TaskCounts(pipelineOperationsTestUtils.getPipelineTaskDisplayData()).toString()); } - @Test - public void testGetTotalSubTaskTotalCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getTotalSubtaskCount()); + public static void testTaskCounts(int taskCount, int waitingToRunTaskCount, + int completedTaskCount, int failedTaskCount, TaskCounts counts) { + assertEquals(taskCount, counts.getTaskCount()); + assertEquals(waitingToRunTaskCount, counts.getTotalCounts().getWaitingToRunTaskCount()); + assertEquals(completedTaskCount, counts.getTotalCounts().getCompletedTaskCount()); + assertEquals(failedTaskCount, counts.getTotalCounts().getFailedTaskCount()); } - @Test - public void testGetTotalSubTaskCompleteCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getCompletedSubtaskCount()); + public static void testCounts(int waitingToRunTaskCount, int processingTaskCount, + int completedTaskCount, int failedTaskCount, int totalSubtaskCount, + int completedSubtaskCount, int failedSubtaskCount, Counts counts) { + assertEquals(waitingToRunTaskCount, counts.getWaitingToRunTaskCount()); + assertEquals(processingTaskCount, counts.getProcessingTaskCount()); + assertEquals(completedTaskCount, counts.getCompletedTaskCount()); + assertEquals(failedTaskCount, counts.getFailedTaskCount()); + assertEquals(totalSubtaskCount, counts.getTotalSubtaskCount()); + assertEquals(completedSubtaskCount, counts.getCompletedSubtaskCount()); + assertEquals(failedSubtaskCount, counts.getFailedSubtaskCount()); } - @Test - public void testGetTotalSubTaskFailedCount() { - TaskCounts taskCounts = new TaskCounts(); - assertEquals(0, taskCounts.getTotalCounts().getFailedSubtaskCount()); + public static void testSubtaskCounts(int totalSubtaskCount, int completedSubtaskCount, + int failedSubtaskCount, SubtaskCounts counts) { + assertEquals(totalSubtaskCount, counts.getTotalSubtaskCount()); + assertEquals(completedSubtaskCount, counts.getCompletedSubtaskCount()); + assertEquals(failedSubtaskCount, counts.getFailedSubtaskCount()); + } + + public static void testTotalSubtaskCounts(int totalSubtaskCount, int completedSubtaskCount, + int failedSubtaskCount, TaskCounts counts) { + assertEquals(totalSubtaskCount, counts.getTotalCounts().getTotalSubtaskCount()); + assertEquals(completedSubtaskCount, counts.getTotalCounts().getCompletedSubtaskCount()); + assertEquals(failedSubtaskCount, counts.getTotalCounts().getFailedSubtaskCount()); } } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperationsTest.java index 9d921a5..aa2ce30 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/GroupOperationsTest.java @@ -11,8 +11,6 @@ import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; /** * Unit tests for {@link GroupOperations}. @@ -21,9 +19,8 @@ */ public class GroupOperationsTest { - // TODO Fix per ZIGGY-431 - // These tests, like the code they are testing, are non-sensical to me. If you ask for groups - // for PipelineDefinition and there are none, you should get none. + private static final String PARAMETER_SET = "ParameterSet"; + private static final String PIPELINE_DEFINITION = "PipelineDefinition"; private GroupOperations groupOperations = new GroupOperations(); @@ -33,18 +30,17 @@ public class GroupOperationsTest { @Test public void testPersist() { Group group1 = createGroup1(); - assertEquals(group1, groupOperations.groupForName("group1", ParameterSet.class)); + assertEquals(group1, groupOperations.group("group1", PARAMETER_SET)); } @Test public void testMerge() { Group group1 = createGroup1(); - assertEquals(group1, groupOperations.groupForName("group1", ParameterSet.class)); + assertEquals(group1, groupOperations.group("group1", PARAMETER_SET)); group1.setName("newGroup"); groupOperations.merge(group1); - assertEquals("newGroup", - groupOperations.groupForName("newGroup", ParameterSet.class).getName()); + assertEquals("newGroup", groupOperations.group("newGroup", PARAMETER_SET).getName()); } @Test @@ -78,52 +74,50 @@ public void testGroups() { } @Test - public void testGroupsForClass() { + public void testGroupsForType() { Group group1 = createGroup1(); - List groups = groupOperations.groupsForClass(ParameterSet.class); + List groups = groupOperations.groups(PARAMETER_SET); assertEquals(1, groups.size()); assertEquals(group1, groups.get(0)); - assertEquals(Set.of("parameterSet1"), groups.get(0).getMemberNames()); + assertEquals(Set.of("parameterSet1"), groups.get(0).getItems()); - groups = groupOperations.groupsForClass(PipelineDefinition.class); - assertEquals(1, groups.size()); - assertEquals(group1, groups.get(0)); - assertEquals(Set.of(), groups.get(0).getMemberNames()); + groups = groupOperations.groups(PIPELINE_DEFINITION); + assertEquals(0, groups.size()); } @Test public void testGroupForName() { - assertTrue(groupOperations.groupForName(null, ParameterSet.class) == Group.DEFAULT); + assertTrue(groupOperations.group(null, PARAMETER_SET) == Group.DEFAULT); - assertTrue(groupOperations.groupForName(" ", ParameterSet.class) == Group.DEFAULT); + assertTrue(groupOperations.group(" ", PARAMETER_SET) == Group.DEFAULT); createGroup1(); - Group group = groupOperations.groupForName("group1", ParameterSet.class); - assertEquals(Set.of("parameterSet1"), group.getMemberNames()); + Group group = groupOperations.group("group1", PARAMETER_SET); + assertEquals(Set.of("parameterSet1"), group.getItems()); - group = groupOperations.groupForName("group1", PipelineDefinition.class); - assertEquals(Set.of(), group.getMemberNames()); + assertTrue(groupOperations.group("group1", PIPELINE_DEFINITION) == Group.DEFAULT); + assertTrue(groupOperations.group("group2", PARAMETER_SET) == Group.DEFAULT); } @Test public void testDelete() { Group group1 = createGroup1(); - assertEquals(group1, groupOperations.groupForName("group1", ParameterSet.class)); + assertEquals(group1, groupOperations.group("group1", PARAMETER_SET)); groupOperations.delete(group1); assertEquals(0, groupOperations.groups().size()); } private Group createGroup1() { - Group group1 = new Group("group1"); - group1.getParameterSetNames().add("parameterSet1"); + Group group1 = new Group("group1", PARAMETER_SET); + group1.getItems().add("parameterSet1"); groupOperations.persist(group1); return group1; } private Group createGroup2() { - Group group2 = new Group("group2"); - group2.getPipelineDefinitionNames().add("pipeline1"); + Group group2 = new Group("group2", PIPELINE_DEFINITION); + group2.getItems().add("pipeline1"); groupOperations.persist(group2); return group2; } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperationsTest.java index b503757..2c73c86 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/ParametersOperationsTest.java @@ -12,7 +12,7 @@ import java.util.Map; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrudTest.java index 3e69e20..81ec703 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineDefinitionCrudTest.java @@ -443,40 +443,6 @@ private void editPipelineDefDeleteLastNode(PipelineDefinition pipelineDef) { nextNodes.clear(); } - @Test - public void testEditPipelineDefinitionDeleteAllNodes() throws Exception { - // Create - populateObjects(); - - // Retrieve & Edit - testOperations.deleteAllNodes(TEST_PIPELINE_NAME_1); - - // Retrieve - PipelineDefinition actualPipelineDef = testOperations - .pipelineDefinition(TEST_PIPELINE_NAME_1); - - PipelineDefinition expectedPipelineDef = createPipelineDefinition(); - editPipelineDefDeleteAllNodes(expectedPipelineDef); - setOptimisticLockValue(expectedPipelineDef, 1); - - comparer.excludeField(".*\\.id"); - - comparer.assertEquals("PipelineDefinition", expectedPipelineDef, actualPipelineDef); - - assertEquals("PipelineDefinitionNode count", 0, pipelineNodeCount()); - assertEquals("PipelineModuleDefinition count", 3, pipelineModuleDefinitionCount()); - assertEquals("ParameterSet count", 1, pipelineModuleParamSetCount()); - } - - /** - * simulate modifications made by a user delete all nodes - * - * @param pipelineDef - */ - private void editPipelineDefDeleteAllNodes(PipelineDefinition pipelineDef) { - pipelineDefinitionCrud.deleteAllPipelineNodes(pipelineDef); - } - @Test public void testRetrievePipelineDefinitionNamesInUse() throws Exception { // No pipeline definitions at all. Should be empty. @@ -611,15 +577,6 @@ public void deleteLastNode(String pipelineDefinitionName) { }); } - public void deleteAllNodes(String pipelineDefinitionName) { - performTransaction(() -> { - PipelineDefinition pipelineDefinition = new PipelineDefinitionCrud() - .retrieveLatestVersionForName(pipelineDefinitionName); - editPipelineDefDeleteAllNodes(pipelineDefinition); - new PipelineDefinitionCrud().merge(pipelineDefinition); - }); - } - public void lockPipelineDefinition(String pipelineDefinitionName) { performTransaction(() -> { PipelineDefinition pipelineDefinition = new PipelineDefinitionCrud() diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperationsTest.java index 717c4e2..8e69fc9 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceNodeOperationsTest.java @@ -8,7 +8,7 @@ import java.util.List; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -24,7 +24,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; import gov.nasa.ziggy.services.database.DatabaseOperations; +import gov.nasa.ziggy.uow.UnitOfWork; /** Unit tests for {@link PipelineInstanceNodeOperations}. */ public class PipelineInstanceNodeOperationsTest { @@ -37,6 +39,8 @@ public class PipelineInstanceNodeOperationsTest { private PipelineInstanceNode pipelineInstanceNode; private PipelineInstanceNode newInstanceNode; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations; private PipelineInstanceOperations pipelineInstanceOperations; @Rule @@ -47,6 +51,8 @@ public void setUp() { pipelineInstanceNodeOperations = Mockito.spy(PipelineInstanceNodeOperations.class); testOperations = new TestOperations(); pipelineTaskOperations = new PipelineTaskOperations(); + pipelineTaskDataOperations = new PipelineTaskDataOperations(); + pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); pipelineInstanceOperations = new PipelineInstanceOperations(); } @@ -89,59 +95,56 @@ public void testMultipleInstanceNodesNoErrors() { // Mark the first 2 tasks as complete and check that all the correct states are // generated. Then create 2 new pipeline tasks in the second instance node. - pipelineTaskOperations.updateProcessingStep(task1, ProcessingStep.COMPLETE); - PipelineTask updatedTask = pipelineTaskOperations.updateProcessingStep(task2, - ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task1, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); PipelineInstance pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task1.getPipelineInstanceId()); assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); TaskCounts counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); - - task3 = new PipelineTask(pipelineInstance, newInstanceNode); - task4 = new PipelineTask(pipelineInstance, newInstanceNode); - List pipelineTasks = testOperations.mergePipelineTasks(List.of(task3, task4)); - for (PipelineTask pipelineTask : pipelineTasks) { - newInstanceNode.addPipelineTask(pipelineTask); - } - newInstanceNode = testOperations.merge(newInstanceNode); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); + + List unitsOfWork = List.of(new UnitOfWork("brief3"), new UnitOfWork("brief4")); + List pipelineTasks = testOperations + .mergePipelineTasks(new RuntimeObjectFactory().newPipelineTasks(newInstanceNode, + pipelineInstance, unitsOfWork)); + task3 = pipelineTasks.get(0); task4 = pipelineTasks.get(1); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 2, 2, 0, counts); + TaskCountsTest.testTaskCounts(4, 2, 2, 0, counts); - counts = pipelineInstanceNodeOperations.taskCounts(newInstanceNode); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); + counts = pipelineTaskDisplayDataOperations.taskCounts(newInstanceNode); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); // Move the new tasks to the EXECUTING step. - pipelineTaskOperations.updateProcessingStep(task3, ProcessingStep.EXECUTING); - updatedTask = pipelineTaskOperations.updateProcessingStep(task4, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task3, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task4, ProcessingStep.EXECUTING); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task3.getPipelineInstanceId()); assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 2, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + TaskCountsTest.testTaskCounts(4, 0, 2, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task3)); + TaskCountsTest.testTaskCounts(2, 0, 0, 0, counts); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); // Move the tasks to COMPLETED. - pipelineTaskOperations.updateProcessingStep(task3, ProcessingStep.COMPLETE); - updatedTask = pipelineTaskOperations.updateProcessingStep(task4, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task3, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task4, ProcessingStep.COMPLETE); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task4.getPipelineInstanceId()); assertEquals(PipelineInstance.State.COMPLETED, pipelineInstance.getState()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 4, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() <= 0); + TaskCountsTest.testTaskCounts(4, 0, 4, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task4)); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + assertFalse(pipelineInstance.getExecutionClock().isRunning()); } @Test @@ -234,16 +237,15 @@ public void testTaskCounts() { pipelineOperationsTestUtils.setUpSingleModulePipeline(); assertEquals(1, pipelineOperationsTestUtils.getPipelineInstanceNodes().size()); - TaskCounts taskCounts = pipelineInstanceNodeOperations + TaskCounts taskCounts = pipelineTaskDisplayDataOperations .taskCounts(pipelineOperationsTestUtils.getPipelineInstanceNodes().get(0)); assertEquals(1, taskCounts.getModuleNames().size()); assertEquals("module1", taskCounts.getModuleNames().get(0)); assertEquals(1, taskCounts.getModuleCounts().size()); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, taskCounts); - PipelineOperationsTestUtils.testCounts(2, 0, 0, 0, 0, 0, 0, - taskCounts.getModuleCounts().get("module1")); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, taskCounts); + TaskCountsTest.testCounts(2, 0, 0, 0, 0, 0, 0, taskCounts.getModuleCounts().get("module1")); - taskCounts = pipelineInstanceNodeOperations + taskCounts = pipelineTaskDisplayDataOperations .taskCounts(pipelineOperationsTestUtils.getPipelineInstanceNodes()); } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperationsTest.java index 3b66b65..852db04 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceOperationsTest.java @@ -7,7 +7,7 @@ import java.util.List; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.hibernate.Hibernate; import org.junit.Before; import org.junit.Rule; diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceTaskCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceTaskCrudTest.java index 6ca5f33..34e01d6 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceTaskCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineInstanceTaskCrudTest.java @@ -32,8 +32,10 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTask_; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData_; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations.ClearStaleStateResults; import gov.nasa.ziggy.services.database.DatabaseOperations; import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; @@ -49,11 +51,14 @@ * * @author Todd Klaus * @author PT + * @author Bill Wohler */ @Category(IntegrationTestCategory.class) public class PipelineInstanceTaskCrudTest { private static final String TEST_PIPELINE_NAME = "Test Pipeline"; + // TODO Remove annotation when constant used in modifyPipelineTask() + @SuppressWarnings("unused") private static final String TEST_WORKER_NAME = "TestWorker"; private PipelineDefinitionCrud pipelineDefinitionCrud; @@ -63,6 +68,7 @@ public class PipelineInstanceTaskCrudTest { private PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud; private ParameterSetCrud parameterSetCrud; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; private PipelineInstanceOperations pipelineInstanceOperations; private PipelineInstance pipelineInstance; @@ -93,6 +99,7 @@ public void setUp() { pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrud(); parameterSetCrud = new ParameterSetCrud(); pipelineTaskOperations = new PipelineTaskOperations(); + pipelineTaskDataOperations = new PipelineTaskDataOperations(); pipelineInstanceOperations = new PipelineInstanceOperations(); testOperations = new TestOperations(); } @@ -110,16 +117,28 @@ private PipelineInstanceNode createPipelineInstanceNode(PipelineDefinitionNode p private PipelineTask createPipelineTask(PipelineInstanceNode parentPipelineInstanceNode) throws PipelineException { - PipelineTask pipelineTask = new PipelineTask(pipelineInstance, parentPipelineInstanceNode); - UnitOfWork uow = PipelineExecutor.generateUnitsOfWork(new SingleUnitOfWorkGenerator(), null) + UnitOfWork unitOfWork = PipelineExecutor + .generateUnitsOfWork(new SingleUnitOfWorkGenerator(), null) .get(0); - pipelineTask.setUowTaskParameters(uow.getParameters()); - pipelineTask.setWorkerHost(TEST_WORKER_NAME); - pipelineTask.setSoftwareRevision("42"); + PipelineTask pipelineTask = new PipelineTask(pipelineInstance, parentPipelineInstanceNode, + unitOfWork); parentPipelineInstanceNode.addPipelineTask(pipelineTask); return pipelineTask; } + private void modifyPipelineTask(PipelineTask pipelineTask, ProcessingStep processingStep) { + modifyPipelineTask(pipelineTask, processingStep, false); + } + + private void modifyPipelineTask(PipelineTask pipelineTask, ProcessingStep processingStep, + boolean error) { + pipelineTaskDataOperations.createPipelineTaskData(pipelineTask, processingStep); + pipelineTaskDataOperations.setError(pipelineTask, error); + // TODO Add, when the time comes + // pipelineTask.setWorkerHost(TEST_WORKER_NAME); + // pipelineTask.setSoftwareRevision("42"); + } + // TODO Move transactionless CRUD methods to TestOperations private int pipelineInstanceCount() { PipelineInstanceCrud crud = new PipelineInstanceCrud(); @@ -134,9 +153,9 @@ private int pipelineTaskCount() { } private int pipelineTasksWithErrorsCount() { - PipelineTaskCrud crud = new PipelineTaskCrud(); - return crud.uniqueResult(crud.createZiggyQuery(PipelineTask.class, Long.class) - .column(PipelineTask_.error) + PipelineTaskDataCrud crud = new PipelineTaskDataCrud(); + return crud.uniqueResult(crud.createZiggyQuery(PipelineTaskData.class, Long.class) + .column(PipelineTaskData_.error) .in(true) .count()).intValue(); } @@ -256,8 +275,8 @@ public void testStoreAndRetrieveTask() throws Exception { testOperations.populateObjects(); // Retrieve - PipelineTask actualPipelineTask = testOperations - .retrieveAndInitializePipelineTask(pipelineTask1.getId()); + PipelineTask actualPipelineTask = pipelineTaskOperations + .pipelineTask(pipelineTask1.getId()); ReflectionEquals comparer = new ReflectionEquals(); comparer.excludeField(".*\\.lastChangedTime"); @@ -267,14 +286,6 @@ public void testStoreAndRetrieveTask() throws Exception { assertEquals("PipelineInstance count", 1, pipelineInstanceCount()); assertEquals("PipelineTask count", 4, pipelineTaskCount()); - - List nodeRevisions = pipelineTaskCrud - .distinctSoftwareRevisions(pipelineInstanceNode1); - assertEquals("nodeRevisions count", 1, nodeRevisions.size()); - - List instanceRevisions = pipelineTaskCrud - .distinctSoftwareRevisions(pipelineInstance); - assertEquals("instanceRevisions count", 1, instanceRevisions.size()); } @Test @@ -282,8 +293,8 @@ public void testStoreAndRetrieveTasks() throws Exception { testOperations.populateObjects(); // Retrieve - List actualPipelineTasks = testOperations.retrieveAndInitializePipelineTasks( - Set.of(pipelineTask1.getId(), pipelineTask2.getId())); + List actualPipelineTasks = pipelineTaskOperations + .pipelineTasks(Set.of(pipelineTask1.getId(), pipelineTask2.getId())); List expectedPipelineTasks = new ArrayList<>(); expectedPipelineTasks.add(pipelineTask1); @@ -297,8 +308,8 @@ public void testStoreAndRetrieveTasksEmptyInputSet() throws Exception { testOperations.populateObjects(); // Retrieve - List actualPipelineTasks = testOperations - .retrieveAndInitializePipelineTasks(new HashSet<>()); + List actualPipelineTasks = pipelineTaskOperations + .pipelineTasks(new HashSet<>()); List expectedPipelineTasks = new ArrayList<>(); @@ -314,7 +325,7 @@ public void testStoreAndRetrieveTasksEmptyInputSet() throws Exception { @Test public void testInstanceState() throws Exception { testOperations.populateObjects(); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 1, 1, + TaskCountsTest.testTaskCounts(4, 0, 1, 1, pipelineInstanceOperations.taskCounts(pipelineInstance)); } @@ -368,43 +379,6 @@ private void editPipelineInstance(PipelineInstance pipelineInstance) { pipelineInstance.setState(PipelineInstance.State.COMPLETED); } - @Test - public void testEditPipelineTask() throws Exception { - // Create - testOperations.populateObjects(); - - // Retrieve & Edit - testOperations.retrieveAndEditPipelineTask(pipelineTask1.getId()); - - // Retrieve - PipelineTask actualPipelineTask = testOperations - .retrieveAndInitializePipelineTask(pipelineTask1.getId()); - - PipelineTask expectedPipelineTask = createPipelineTask(pipelineInstanceNode1); - editPipelineTask(expectedPipelineTask); - - ReflectionEquals comparer = new ReflectionEquals(); - comparer.excludeField(".*\\.id"); - comparer.excludeField(".*\\.created"); - comparer.excludeField(".*\\.lastChangedTime"); - comparer.excludeField(".*\\.classname"); - comparer.excludeField(".*\\.xmlParameters"); - - comparer.assertEquals("PipelineTask", expectedPipelineTask, actualPipelineTask); - - assertEquals("PipelineInstance count", 1, pipelineInstanceCount()); - assertEquals("PipelineTask count", 4, pipelineTaskCount()); - } - - /** - * simulate modifications made by a user - * - * @param pipelineDef - */ - private void editPipelineTask(PipelineTask pipelineTask) { - pipelineTask.setProcessingStep(ProcessingStep.COMPLETE); - } - @Test public void testClearStaleState() throws Exception { // Create @@ -423,7 +397,7 @@ public void testClearStaleState() throws Exception { assertEquals("unique instance id", pipelineInstance.getId(), staleStateResults.uniqueInstanceIds.iterator().next()); assertEquals("pipelineTaskWithErrorsCount count", 3, pipelineTasksWithErrorsCount()); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 1, 3, + TaskCountsTest.testTaskCounts(4, 0, 1, 3, pipelineInstanceOperations.taskCounts(pipelineInstance)); } @@ -509,35 +483,28 @@ public void populateObjects() { pipelineInstance = pipelineInstanceCrud.merge(createPipelineInstance()); pipelineInstanceNode1 = pipelineInstanceNodeCrud .merge(createPipelineInstanceNode(pipelineDefNode1)); - pipelineInstanceNode2 = pipelineInstanceNodeCrud - .merge(createPipelineInstanceNode(pipelineDefNode2)); + pipelineInstance.addPipelineInstanceNode(pipelineInstanceNode1); + pipelineInstance = pipelineInstanceCrud.merge(pipelineInstance); pipelineTask1 = createPipelineTask(pipelineInstanceNode1); - pipelineTask1.setProcessingStep(ProcessingStep.EXECUTING); pipelineTaskCrud.persist(pipelineTask1); + modifyPipelineTask(pipelineTask1, ProcessingStep.EXECUTING); pipelineTask2 = createPipelineTask(pipelineInstanceNode1); - pipelineTask2.setProcessingStep(ProcessingStep.COMPLETE); pipelineTaskCrud.persist(pipelineTask2); + modifyPipelineTask(pipelineTask2, ProcessingStep.COMPLETE); - pipelineInstanceNode2 = createPipelineInstanceNode(pipelineDefNode2); - pipelineInstanceNodeCrud.persist(pipelineInstanceNode2); + pipelineInstanceNode2 = pipelineInstanceNodeCrud + .merge(createPipelineInstanceNode(pipelineDefNode2)); + pipelineInstance.addPipelineInstanceNode(pipelineInstanceNode2); pipelineTask3 = createPipelineTask(pipelineInstanceNode2); - pipelineTask3.setProcessingStep(ProcessingStep.EXECUTING); pipelineTaskCrud.persist(pipelineTask3); + modifyPipelineTask(pipelineTask3, ProcessingStep.EXECUTING); pipelineTask4 = createPipelineTask(pipelineInstanceNode2); - pipelineTask4.setProcessingStep(ProcessingStep.EXECUTING); - pipelineTask4.setError(true); pipelineTaskCrud.persist(pipelineTask4); - - pipelineInstanceNode1 = pipelineInstanceNodeCrud.merge(pipelineInstanceNode1); - pipelineInstanceNode2 = pipelineInstanceNodeCrud.merge(pipelineInstanceNode2); - - pipelineInstance.addPipelineInstanceNode(pipelineInstanceNode1); - pipelineInstance.addPipelineInstanceNode(pipelineInstanceNode2); - pipelineInstanceCrud.merge(pipelineInstance); + modifyPipelineTask(pipelineTask4, ProcessingStep.EXECUTING, true); }); } @@ -579,31 +546,6 @@ public List retrieveAndInitializeAllPipelineInstanceNodes( }); } - public PipelineTask retrieveAndInitializePipelineTask(long pipelineTaskId) { - return performTransaction(() -> { - PipelineTask pi = pipelineTaskCrud.retrieve(pipelineTaskId); - ZiggyUnitTestUtils.initializePipelineTask(pi); - return pi; - }); - } - - public List retrieveAndInitializePipelineTasks(Collection taskIds) { - return performTransaction(() -> { - List tasks = pipelineTaskCrud.retrieveAll(taskIds); - for (PipelineTask task : tasks) { - ZiggyUnitTestUtils.initializePipelineTask(task); - } - return tasks; - }); - } - - public void retrieveAndEditPipelineTask(long pipelineTaskId) { - performTransaction(() -> { - PipelineTask pTask = pipelineTaskCrud.retrieve(pipelineTaskId); - editPipelineTask(pTask); - }); - } - public PipelineInstance setPipelineInstanceEndNode(long pipelineInstanceId, PipelineInstanceNode endNode) { return performTransaction(() -> { diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineOperationsTestUtils.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineOperationsTestUtils.java index 23adfff..ed3db98 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineOperationsTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineOperationsTestUtils.java @@ -2,9 +2,12 @@ import static com.google.common.base.Preconditions.checkState; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; +import java.util.Set; import org.mockito.Mockito; @@ -14,10 +17,15 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.TaskCounts; -import gov.nasa.ziggy.pipeline.definition.TaskCounts.Counts; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.pipeline.definition.TaskCounts.SubtaskCounts; import gov.nasa.ziggy.services.database.DatabaseOperations; +import gov.nasa.ziggy.uow.UnitOfWork; import gov.nasa.ziggy.util.ZiggyCollectionUtils; /** @@ -34,7 +42,8 @@ public class PipelineOperationsTestUtils extends DatabaseOperations { private List pipelineInstances; private List pipelineInstanceNodes; private List pipelineTasks; - private PipelineTask task1, task2; + private List pipelineTaskDataList; + private List pipelineTaskDisplayData; /** * Generates and persists a pipeline definition with a single module and two tasks. Its name is @@ -94,20 +103,14 @@ public void generateSingleModulePipeline(boolean persist) { () -> new PipelineModuleDefinitionCrud().merge(pipelineModuleDefinitions.get(0)))); // Persist the pipeline tasks and update the pipeline instance node. - task1 = new PipelineTask(pipelineInstances.get(0), pipelineInstanceNodes.get(0)); - task2 = new PipelineTask(pipelineInstances.get(0), pipelineInstanceNodes.get(0)); - pipelineTasks = performTransaction(() -> { - PipelineTask mergedTask1 = new PipelineTaskCrud().merge(task1); - PipelineTask mergedTask2 = new PipelineTaskCrud().merge(task2); - return List.of(mergedTask1, mergedTask2); - }); - pipelineInstanceNodes.get(0).addPipelineTask(pipelineTasks.get(0)); - pipelineInstanceNodes.get(0).addPipelineTask(pipelineTasks.get(1)); - pipelineInstanceNodes = ZiggyCollectionUtils.mutableListOf(performTransaction( - () -> new PipelineInstanceNodeCrud().merge(pipelineInstanceNodes.get(0)))); + List unitsOfWork = List.of(new UnitOfWork("brief0"), + new UnitOfWork("brief1")); + pipelineTasks = new RuntimeObjectFactory().newPipelineTasks( + pipelineInstanceNodes.get(0), pipelineInstances.get(0), unitsOfWork); + pipelineTaskDataList = new TestOperations().createPipelineTaskData(pipelineTasks); } else { - task1 = new PipelineTask(pipelineInstance, instanceNode); - task2 = new PipelineTask(pipelineInstance, instanceNode); + PipelineTask task1 = new PipelineTask(pipelineInstance, instanceNode, null); + PipelineTask task2 = new PipelineTask(pipelineInstance, instanceNode, null); pipelineModuleDefinitions = ZiggyCollectionUtils.mutableListOf(moduleDefinition); pipelineDefinitions = ZiggyCollectionUtils.mutableListOf(pipelineDefinition); pipelineDefinitionNodes = List.of(definitionNode); @@ -122,6 +125,67 @@ public void generateSingleModulePipeline(boolean persist) { } } + public void testPipelineTaskDisplayData( + List pipelineTaskDisplayDataList) { + for (PipelineTaskDisplayData pipelineTaskDisplayData : pipelineTaskDisplayDataList) { + testPipelineTaskDisplayData(pipelineTaskDisplayData); + } + } + + public void testPipelineTaskDisplayData(PipelineTaskDisplayData pipelineTaskDisplayData) { + // Does this object belong to our instance? + assertEquals((long) pipelineInstance().getId(), + pipelineTaskDisplayData.getPipelineInstanceId()); + + // Does this object belong to one of our tasks? + PipelineTask pipelineTask = null; + for (PipelineTask task : pipelineTasks) { + if (task.getId().longValue() == pipelineTaskDisplayData.getPipelineTaskId()) { + pipelineTask = task; + break; + } + } + assertNotNull(pipelineTask); + + // Does this object belong to one of our task data objects? + PipelineTaskData pipelineTaskData = null; + for (PipelineTaskData taskData : pipelineTaskDataList) { + if (taskData.getPipelineTask().equals(pipelineTaskDisplayData.getPipelineTask())) { + pipelineTaskData = taskData; + break; + } + } + assertNotNull(pipelineTaskData); + + // Carry on, using our saved pipeline tasks and pipeline task data. + assertEquals(pipelineTask.getCreated(), pipelineTaskDisplayData.getCreated()); + assertEquals(pipelineTask.getModuleName(), pipelineTaskDisplayData.getModuleName()); + assertEquals(pipelineTask.getUnitOfWork().briefState(), + pipelineTaskDisplayData.getBriefState()); + + assertEquals(pipelineTaskData.getZiggySoftwareRevision(), + pipelineTaskDisplayData.getZiggySoftwareRevision()); + assertEquals(pipelineTaskData.getPipelineSoftwareRevision(), + pipelineTaskDisplayData.getPipelineSoftwareRevision()); + assertEquals("host" + pipelineTask.getId() + ":" + pipelineTask.getId(), + pipelineTaskDisplayData.getWorkerName()); + assertEquals(pipelineTaskData.getProcessingStep(), + pipelineTaskDisplayData.getProcessingStep()); + assertEquals(pipelineTaskData.isError(), pipelineTaskDisplayData.isError()); + assertEquals(pipelineTaskData.getTotalSubtaskCount(), + pipelineTaskDisplayData.getTotalSubtaskCount()); + assertEquals(pipelineTaskData.getCompletedSubtaskCount(), + pipelineTaskDisplayData.getCompletedSubtaskCount()); + assertEquals(pipelineTaskData.getFailedSubtaskCount(), + pipelineTaskDisplayData.getFailedSubtaskCount()); + assertEquals(pipelineTaskData.getFailureCount(), pipelineTaskDisplayData.getFailureCount()); + assertEquals(pipelineTaskData.getExecutionClock().toString(), + pipelineTaskDisplayData.getExecutionClock().toString()); + assertEquals(pipelineTaskData.getPipelineTaskMetrics(), + pipelineTaskDisplayData.getPipelineTaskMetrics()); + assertEquals(pipelineTaskData.getRemoteJobs(), pipelineTaskDisplayData.getRemoteJobs()); + } + /** * Generates and persists a pipeline definition with a four modules and no tasks. The names are * module1 through module4 and its pipeline definition's name is pipeline1. @@ -311,17 +375,56 @@ private PipelineTask pipelineTask(String moduleName, Long id, int attributeSeed, ProcessingStep processingStep, boolean error) { PipelineInstanceNode pipelineInstanceNode = new PipelineInstanceNode(); pipelineInstanceNode.setPipelineModuleDefinition(new PipelineModuleDefinition(moduleName)); - PipelineTask pipelineTask = Mockito.spy(new PipelineTask(null, pipelineInstanceNode)); + PipelineTask pipelineTask = Mockito.spy(new PipelineTask(null, pipelineInstanceNode, null)); Mockito.doReturn(id).when(pipelineTask).getId(); - pipelineTask.setTotalSubtaskCount(attributeSeed); - pipelineTask.setCompletedSubtaskCount(attributeSeed - (int) (0.1 * attributeSeed)); - pipelineTask.setFailedSubtaskCount((int) (0.1 * attributeSeed)); - pipelineTask.setProcessingStep(processingStep); - pipelineTask.setError(error); + SubtaskCounts subtaskCounts = new SubtaskCounts(attributeSeed, + attributeSeed - (int) (0.1 * attributeSeed), (int) (0.1 * attributeSeed)); + Mockito.when(Mockito.spy(new PipelineTaskDataOperations()).subtaskCounts(pipelineTask)) + .thenReturn(subtaskCounts); + // pipelineTask.setProcessingStep(processingStep); + // pipelineTask.setError(error); return pipelineTask; } + public void setUpFivePipelineTaskDisplayData() { + pipelineTaskDisplayData = new ArrayList<>(); + pipelineTaskDisplayData + .add(pipelineTaskDisplayData("module1", 1L, 10, ProcessingStep.INITIALIZING)); + pipelineTaskDisplayData + .add(pipelineTaskDisplayData("module2", 2L, 20, ProcessingStep.WAITING_TO_RUN)); + pipelineTaskDisplayData + .add(pipelineTaskDisplayData("module3", 3L, 30, ProcessingStep.EXECUTING)); + pipelineTaskDisplayData + .add(pipelineTaskDisplayData("module4", 4L, 40, ProcessingStep.EXECUTING, true)); + pipelineTaskDisplayData + .add(pipelineTaskDisplayData("module5", 5L, 50, ProcessingStep.COMPLETE)); + } + + private PipelineTaskDisplayData pipelineTaskDisplayData(String moduleName, Long id, + int attributeSeed, ProcessingStep processingStep) { + return pipelineTaskDisplayData(moduleName, id, attributeSeed, processingStep, false); + } + + private PipelineTaskDisplayData pipelineTaskDisplayData(String moduleName, Long id, + int attributeSeed, ProcessingStep processingStep, boolean error) { + PipelineInstanceNode pipelineInstanceNode = new PipelineInstanceNode(); + pipelineInstanceNode.setPipelineModuleDefinition(new PipelineModuleDefinition(moduleName)); + UnitOfWork unitOfWork = new UnitOfWork(moduleName); + PipelineTask pipelineTask = Mockito + .spy(new PipelineTask(null, pipelineInstanceNode, unitOfWork)); + Mockito.doReturn(id).when(pipelineTask).getId(); + + PipelineTaskData pipelineTaskData = new PipelineTaskData(pipelineTask); + pipelineTaskData.setProcessingStep(processingStep); + pipelineTaskData.setTotalSubtaskCount(attributeSeed); + pipelineTaskData.setCompletedSubtaskCount(attributeSeed - (int) (0.1 * attributeSeed)); + pipelineTaskData.setFailedSubtaskCount((int) (0.1 * attributeSeed)); + pipelineTaskData.setError(error); + + return new PipelineTaskDisplayData(pipelineTaskData); + } + public List getPipelineModuleDefinitions() { return pipelineModuleDefinitions; } @@ -346,6 +449,10 @@ public List getPipelineTasks() { return pipelineTasks; } + public List getPipelineTaskDisplayData() { + return pipelineTaskDisplayData; + } + // Special-case getters for situations in which there is one and only one element // in a list. The "get" prefix is not used because these do not get a field of the // instance. @@ -375,23 +482,37 @@ public PipelineInstanceNode pipelineInstanceNode() { return pipelineInstanceNodes.get(0); } - public static void testTaskCounts(int taskCount, int waitingToRunTaskCount, - int completedTaskCount, int failedTaskCount, TaskCounts counts) { - assertEquals(taskCount, counts.getTaskCount()); - assertEquals(waitingToRunTaskCount, counts.getTotalCounts().getWaitingToRunTaskCount()); - assertEquals(completedTaskCount, counts.getTotalCounts().getCompletedTaskCount()); - assertEquals(failedTaskCount, counts.getTotalCounts().getFailedTaskCount()); - } + private static class TestOperations extends DatabaseOperations { + PipelineTaskDataCrud pipelineTaskDataCrud = new PipelineTaskDataCrud(); + + private List createPipelineTaskData(List pipelineTasks) { + List pipelineTaskDataList = new ArrayList<>(); + + for (PipelineTask pipelineTask : pipelineTasks) { + PipelineTaskData pipelineTaskData = pipelineTaskDataCrud + .retrievePipelineTaskData(pipelineTask); - public static void testCounts(int waitingToRunTaskCount, int processingTaskCount, - int completedTaskCount, int failedTaskCount, int totalSubtaskCount, - int completedSubtaskCount, int failedSubtaskCount, Counts counts) { - assertEquals(waitingToRunTaskCount, counts.getWaitingToRunTaskCount()); - assertEquals(processingTaskCount, counts.getProcessingTaskCount()); - assertEquals(completedTaskCount, counts.getCompletedTaskCount()); - assertEquals(failedTaskCount, counts.getFailedTaskCount()); - assertEquals(totalSubtaskCount, counts.getTotalSubtaskCount()); - assertEquals(completedSubtaskCount, counts.getCompletedSubtaskCount()); - assertEquals(failedSubtaskCount, counts.getFailedSubtaskCount()); + pipelineTaskData.setPipelineTaskMetrics( + createPipelineTaskMetrics(pipelineTask.getModuleName())); + pipelineTaskData.setRemoteJobs(createRemoteJobs(pipelineTask)); + pipelineTaskData.setZiggySoftwareRevision("ziggy software revision 1"); + pipelineTaskData.setPipelineSoftwareRevision("pipeline software revision 1"); + pipelineTaskData.setWorkerHost("host" + pipelineTask.getId()); + pipelineTaskData.setWorkerThread(pipelineTask.getId().intValue()); + + pipelineTaskDataList.add(pipelineTaskDataCrud.merge(pipelineTaskData)); + } + + return pipelineTaskDataList; + } + + private List createPipelineTaskMetrics(String moduleName) { + return new ArrayList<>(List.of(new PipelineTaskMetric(moduleName, 42, Units.TIME))); + } + + private Set createRemoteJobs(PipelineTask pipelineTask) { + return new HashSet<>(Set.of(new RemoteJob(10 * pipelineTask.getId()), + new RemoteJob(20 * pipelineTask.getId()))); + } } } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrudTest.java index af300f4..10474ab 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskCrudTest.java @@ -9,7 +9,7 @@ import java.util.stream.Collectors; import java.util.stream.Stream; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -169,7 +169,7 @@ private PipelineInstance createTasksForPipeline(String instanceName, instance.addPipelineInstanceNode(new PipelineInstanceCrud().merge(instanceNode)); for (int i = 0; i < taskCount; i++) { - PipelineTask task = new PipelineTask(instance, instanceNode); + PipelineTask task = new PipelineTask(instance, instanceNode, null); instanceNode.addPipelineTask(new PipelineTaskCrud().merge(task)); } instanceNode = new PipelineInstanceNodeCrud().merge(instanceNode); @@ -209,10 +209,6 @@ protected void resumeCurrentStep() { protected void resubmit() { } - @Override - protected void resumeMonitoring() { - } - @Override protected List restartModes() { return null; diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperationsTest.java new file mode 100644 index 0000000..d1334cb --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDataOperationsTest.java @@ -0,0 +1,460 @@ +package gov.nasa.ziggy.pipeline.definition.database; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Function; +import java.util.stream.Collectors; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyDatabaseRule; +import gov.nasa.ziggy.module.AlgorithmType; +import gov.nasa.ziggy.module.remote.QueueCommandManager; +import gov.nasa.ziggy.module.remote.QueueCommandManagerTest; +import gov.nasa.ziggy.module.remote.RemoteJobInformation; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetric.Units; +import gov.nasa.ziggy.pipeline.definition.ProcessingStep; +import gov.nasa.ziggy.pipeline.definition.RemoteJob; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; +import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; + +public class PipelineTaskDataOperationsTest { + + @Rule + public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); + + private PipelineTaskOperations pipelineTaskOperations = new PipelineTaskOperations(); + private PipelineTaskDataOperations pipelineTaskDataOperations; + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations; + private PipelineTaskOperationsTest pipelineTaskOperationsTest; + private PipelineOperationsTestUtils pipelineOperationsTestUtils; + private QueueCommandManager cmdManager; + + @Before + public void setUp() { + pipelineTaskOperationsTest = new PipelineTaskOperationsTest(); + pipelineTaskOperationsTest.setUp(); + cmdManager = spy(QueueCommandManager.newInstance()); + pipelineTaskDataOperations = Mockito.spy(PipelineTaskDataOperations.class); + when(pipelineTaskDataOperations.queueCommandManager()).thenReturn(cmdManager); + pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + + pipelineOperationsTestUtils = new PipelineOperationsTestUtils(); + pipelineOperationsTestUtils.setUpSingleModulePipeline(); + } + + @Test + public void testProcessingStep() { + assertEquals(ProcessingStep.WAITING_TO_RUN, pipelineTaskDataOperations + .processingStep(pipelineOperationsTestUtils.getPipelineTasks().get(0))); + } + + @Test + public void testPipelineTasks() { + assertEquals(pipelineOperationsTestUtils.getPipelineTasks(), + pipelineTaskDataOperations.pipelineTasks(pipelineOperationsTestUtils.pipelineInstance(), + Set.of(ProcessingStep.WAITING_TO_RUN))); + assertEquals(List.of(), pipelineTaskDataOperations.pipelineTasks( + pipelineOperationsTestUtils.pipelineInstance(), Set.of(ProcessingStep.EXECUTING))); + } + + @Test + public void testUpdateProcessingStep() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.EXECUTING); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + assertTrue(pipelineTaskDataOperations.executionClock(pipelineTask).isRunning()); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.COMPLETE); + assertEquals(ProcessingStep.COMPLETE, + pipelineTaskDataOperations.processingStep(pipelineTask)); + assertFalse(pipelineTaskDataOperations.executionClock(pipelineTask).isRunning()); + } + + @Test + public void testError() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.taskErrored(pipelineTask); + assertTrue(pipelineTaskDataOperations.hasErrored(pipelineTask)); + assertEquals(List.of(pipelineTask), pipelineTaskDataOperations + .erroredPipelineTasks(pipelineOperationsTestUtils.pipelineInstance())); + pipelineTaskDataOperations.clearError(pipelineTask); + assertFalse(pipelineTaskDataOperations.hasErrored(pipelineTask)); + pipelineTaskDataOperations.taskErrored(pipelineTask); + assertTrue(pipelineTaskDataOperations.hasErrored(pipelineTask)); + pipelineTaskDataOperations.setError(pipelineTask, false); + assertFalse(pipelineTaskDataOperations.hasErrored(pipelineTask)); + } + + @Test + public void testHaltRequested() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + assertFalse(pipelineTaskDataOperations.haltRequested(pipelineTask)); + pipelineTaskDataOperations.setHaltRequested(pipelineTask, true); + assertTrue(pipelineTaskDataOperations.haltRequested(pipelineTask)); + pipelineTaskDataOperations.setHaltRequested(pipelineTask, false); + assertFalse(pipelineTaskDataOperations.haltRequested(pipelineTask)); + } + + @Test + public void testSetWorkerInfo() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateWorkerInfo(pipelineTask, "host", 42); + + assertEquals("host:42", + pipelineTaskDisplayDataOperations.pipelineTaskDisplayData(pipelineTask) + .getWorkerName()); + } + + @Test + public void testDistinctSoftwareRevisions() { + assertEquals(List.of("ziggy software revision 1", "pipeline software revision 1"), + pipelineTaskDataOperations + .distinctSoftwareRevisions(pipelineOperationsTestUtils.pipelineInstance())); + assertEquals(List.of("ziggy software revision 1", "pipeline software revision 1"), + pipelineTaskDataOperations + .distinctSoftwareRevisions(pipelineOperationsTestUtils.pipelineInstanceNode())); + } + + @Test + public void testPrepareTasksForManualResubmit() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations + .prepareTasksForManualResubmit(pipelineOperationsTestUtils.getPipelineTasks()); + assertEquals(0, pipelineTaskDataOperations.autoResubmitCount(pipelineTask)); + assertTrue(pipelineTaskDataOperations.retrying(pipelineTask)); + } + + @Test + public void testPrepareTaskForAutoResubmit() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.prepareTaskForAutoResubmit(pipelineTask); + assertEquals(1, pipelineTaskDataOperations.autoResubmitCount(pipelineTask)); + assertTrue(pipelineTaskDataOperations.hasErrored(pipelineTask)); + } + + @Test + public void testPrepareTaskForRestart() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.taskErrored(pipelineTask); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + assertTrue(pipelineTaskDataOperations.hasErrored(pipelineTask)); + + pipelineTaskDataOperations.prepareTaskForRestart(pipelineTask); + assertEquals(ProcessingStep.EXECUTING, + pipelineTaskDataOperations.processingStep(pipelineTask)); + assertFalse(pipelineTaskDataOperations.hasErrored(pipelineTask)); + } + + @Test + public void testSubtaskCounts() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, 1, 2, 3); + TaskCountsTest.testSubtaskCounts(1, 2, 3, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, -1, -2, -3); + TaskCountsTest.testSubtaskCounts(1, 2, 3, + pipelineTaskDataOperations.subtaskCounts(pipelineTask)); + } + + @Test + public void testIncrementFailureCount() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + assertEquals(0, pipelineTaskDisplayDataOperations.pipelineTaskDisplayData(pipelineTask) + .getFailureCount()); + pipelineTaskDataOperations.incrementFailureCount(pipelineTask); + assertEquals(1, pipelineTaskDisplayDataOperations.pipelineTaskDisplayData(pipelineTask) + .getFailureCount()); + } + + @Test + public void testAlgorithmType() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateAlgorithmType(pipelineTask, AlgorithmType.REMOTE); + assertEquals(AlgorithmType.REMOTE, pipelineTaskDataOperations.algorithmType(pipelineTask)); + } + + @Test + public void testIncrementTaskLogIndex() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + assertEquals("1-1-module1.42-0.log", + pipelineTaskDataOperations.logFilename(pipelineTask, 42)); + pipelineTaskDataOperations.incrementTaskLogIndex(pipelineTask); + pipelineTaskDataOperations.incrementTaskLogIndex(pipelineTask); + assertEquals("1-1-module1.42-2.log", + pipelineTaskDataOperations.logFilename(pipelineTask, 42)); + assertEquals("1-1-module1.42-24.log", + pipelineTaskDataOperations.logFilename(pipelineTask, 42, 24)); + } + + @Test + public void testPipelineTaskMetrics() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + + List existingPipelineTaskMetrics = pipelineTaskDataOperations + .pipelineTaskMetrics(pipelineTask); + assertEquals(1, existingPipelineTaskMetrics.size()); + assertEquals(42, existingPipelineTaskMetrics.get(0).getValue()); + + Map> taskMetricsByTask = createPipelineTaskMetrics( + List.of(pipelineTask)); + + existingPipelineTaskMetrics = pipelineTaskDataOperations.pipelineTaskMetrics(pipelineTask); + assertEquals(3, existingPipelineTaskMetrics.size()); + Map metricByCategory = existingPipelineTaskMetrics.stream() + .collect(Collectors.toMap(PipelineTaskMetric::getCategory, Function.identity())); + for (PipelineTaskMetric pipelineTaskMetric : taskMetricsByTask.get(pipelineTask)) { + assertEquals(pipelineTaskMetric, + metricByCategory.get(pipelineTaskMetric.getCategory())); + } + + PipelineTaskMetric pipelineTaskMetric = new PipelineTaskMetric("metric4", 400, Units.RATE); + existingPipelineTaskMetrics.add(pipelineTaskMetric); + pipelineTaskDataOperations.updatePipelineTaskMetrics(pipelineTask, + existingPipelineTaskMetrics); + existingPipelineTaskMetrics = pipelineTaskDataOperations.pipelineTaskMetrics(pipelineTask); + assertEquals(4, existingPipelineTaskMetrics.size()); + assertTrue(existingPipelineTaskMetrics.contains(pipelineTaskMetric)); + + pipelineTaskMetric = existingPipelineTaskMetrics.get(0); + existingPipelineTaskMetrics.remove(pipelineTaskMetric); + pipelineTaskDataOperations.updatePipelineTaskMetrics(pipelineTask, + existingPipelineTaskMetrics); + existingPipelineTaskMetrics = pipelineTaskDataOperations.pipelineTaskMetrics(pipelineTask); + assertEquals(3, existingPipelineTaskMetrics.size()); + assertFalse(existingPipelineTaskMetrics.contains(pipelineTaskMetric)); + } + + @Test + public void testTaskMetricsByTask() { + Map> expectedTaskMetricsByTask = createPipelineTaskMetrics( + pipelineOperationsTestUtils.getPipelineTasks()); + + Map> taskMetricsByTask = pipelineTaskDataOperations + .taskMetricsByTask(pipelineOperationsTestUtils.pipelineInstanceNode()); + assertEquals(expectedTaskMetricsByTask, taskMetricsByTask); + } + + private Map> createPipelineTaskMetrics( + List pipelineTasks) { + Map> taskMetricsByTask = new HashMap<>(); + + int pipelineTaskIndex = 100; + for (PipelineTask pipelineTask : pipelineTasks) { + List pipelineTaskMetrics = new ArrayList<>(); + pipelineTaskMetrics + .add(new PipelineTaskMetric("metric1", pipelineTaskIndex + 1, Units.TIME)); + pipelineTaskMetrics + .add(new PipelineTaskMetric("metric2", pipelineTaskIndex + 2, Units.BYTES)); + pipelineTaskMetrics + .add(new PipelineTaskMetric("metric3", pipelineTaskIndex + 3, Units.RATE)); + taskMetricsByTask.put(pipelineTask, pipelineTaskMetrics); + pipelineTaskDataOperations.updatePipelineTaskMetrics(pipelineTask, pipelineTaskMetrics); + pipelineTaskIndex += 100; + } + return taskMetricsByTask; + } + + @Test + public void testTaskExecutionLogs() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + List taskExecutionLogs = pipelineTaskDataOperations + .taskExecutionLogs(pipelineTask); + assertEquals(0, taskExecutionLogs.size()); + + pipelineTaskDataOperations.addTaskExecutionLog(pipelineTask, "dummy1", 1, 0L); + pipelineTaskDataOperations.addTaskExecutionLog(pipelineTask, "dummy2", 2, 0L); + pipelineTaskDataOperations.addTaskExecutionLog(pipelineTask, "dummy3", 3, 0L); + taskExecutionLogs = pipelineTaskDataOperations.taskExecutionLogs(pipelineTask); + assertEquals(3, taskExecutionLogs.size()); + + Map logsByWorkerHost = new HashMap<>(); + for (TaskExecutionLog taskExecutionLog : taskExecutionLogs) { + logsByWorkerHost.put(taskExecutionLog.getWorkerHost(), taskExecutionLog); + } + testTaskExecutionLog(logsByWorkerHost.get("dummy1"), 1); + testTaskExecutionLog(logsByWorkerHost.get("dummy2"), 2); + testTaskExecutionLog(logsByWorkerHost.get("dummy3"), 3); + } + + private void testTaskExecutionLog(TaskExecutionLog taskExecutionLog, int workerThread) { + assertNotNull(taskExecutionLog); + assertEquals(workerThread, taskExecutionLog.getWorkerThread()); + } + + @Test + public void testEmptyRemoteJobs() { + PipelineTask pipelineTask = spy(new PipelineTask(null, new PipelineInstanceNode(), null)); + doReturn(42L).when(pipelineTask).getId(); + + Set remoteJobs = pipelineTaskDataOperations.remoteJobs(pipelineTask); + assertEquals(0, remoteJobs.size()); + } + + @Test + public void testRemoteJobs() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + + Set expectedJobs = new HashSet<>( + Set.of(new RemoteJob(3), new RemoteJob(2), new RemoteJob(1))); + pipelineTaskDataOperations.updateRemoteJobs(pipelineTask, expectedJobs); + Set remoteJobs = pipelineTaskDataOperations.remoteJobs(pipelineTask); + assertEquals(3, remoteJobs.size()); + assertEquals(expectedJobs, remoteJobs); + + PipelineTaskDisplayData pipelineTaskDisplayData = pipelineTaskDisplayDataOperations + .pipelineTaskDisplayData(pipelineTask); + assertEquals(3, pipelineTaskDisplayData.getRemoteJobs().size()); + assertEquals(expectedJobs, pipelineTaskDisplayData.getRemoteJobs()); + } + + @Test + public void testRemoteJobsSort() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + + Set expectedJobs = new HashSet<>( + Set.of(new RemoteJob(3), new RemoteJob(2), new RemoteJob(1))); + pipelineTaskDataOperations.updateRemoteJobs(pipelineTask, expectedJobs); + List remoteJobList = new ArrayList<>( + pipelineTaskDataOperations.remoteJobs(pipelineTask)); + assertEquals(new RemoteJob(1), remoteJobList.get(0)); + assertEquals(new RemoteJob(2), remoteJobList.get(1)); + assertEquals(new RemoteJob(3), remoteJobList.get(2)); + } + + @Test + public void testCreateRemoteJobsFromQstat() { + + PipelineTask task = pipelineTaskOperationsTest.createPipelineTask(); + List remoteJobsInformation = remoteJobsInformation(); + pipelineTaskDataOperations.addRemoteJobs(task, remoteJobsInformation); + + Set remoteJobs = pipelineTaskDataOperations.remoteJobs(task); + assertEquals(3, remoteJobs.size()); + assertTrue(remoteJobs.contains(new RemoteJob(9101154))); + assertTrue(remoteJobs.contains(new RemoteJob(9102337))); + assertTrue(remoteJobs.contains(new RemoteJob(6020203))); + for (RemoteJob job : remoteJobs) { + assertFalse(job.isFinished()); + assertEquals(0, job.getCostEstimate(), 1e-9); + } + } + + @Test + public void testUpdateJobs() { + + // Create the task with 3 remote jobs + PipelineTask task = pipelineTaskOperationsTest.createPipelineTask(); + List remoteJobsInformation = remoteJobsInformation(); + pipelineTaskDataOperations.addRemoteJobs(task, remoteJobsInformation); + + mockRemoteJobUpdates(); + + pipelineTaskDataOperations.updateJobs(task); + + // Check for the expected results + List remoteJobs = new ArrayList<>(pipelineTaskDataOperations.remoteJobs(task)); + RemoteJob job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9101154))); + assertTrue(job.isFinished()); + assertEquals(20.0, job.getCostEstimate(), 1e-9); + job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9102337))); + assertFalse(job.isFinished()); + assertEquals(8.0, job.getCostEstimate(), 1e-9); + job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(6020203))); + assertFalse(job.isFinished()); + assertEquals(0, job.getCostEstimate(), 1e-9); + + // Make sure that the database was also updated + Set databaseRemoteJobs = pipelineTaskDataOperations.remoteJobs(task); + for (RemoteJob remoteJob : databaseRemoteJobs) { + RemoteJob otherJob = remoteJobs.get(remoteJobs.indexOf(remoteJob)); + assertEquals(otherJob.isFinished(), remoteJob.isFinished()); + assertEquals(otherJob.getCostEstimate(), remoteJob.getCostEstimate(), 1e-9); + } + } + + @Test + public void testUpdateJobsForPipelineInstance() { + + // Create the task with 3 remote jobs + PipelineTask task = pipelineTaskOperationsTest.createPipelineTask(); + List remoteJobsInformation = remoteJobsInformation(); + pipelineTaskDataOperations.addRemoteJobs(task, remoteJobsInformation); + + mockRemoteJobUpdates(); + + pipelineTaskDataOperations.updateJobs(pipelineTaskOperations.pipelineInstance(task)); + + // Check for the expected results + List remoteJobs = new ArrayList<>(pipelineTaskDataOperations.remoteJobs(task)); + RemoteJob job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9101154))); + assertTrue(job.isFinished()); + assertEquals(20.0, job.getCostEstimate(), 1e-9); + job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9102337))); + assertFalse(job.isFinished()); + assertEquals(8.0, job.getCostEstimate(), 1e-9); + job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(6020203))); + assertFalse(job.isFinished()); + assertEquals(0, job.getCostEstimate(), 1e-9); + } + + private List remoteJobsInformation() { + List remoteJobsInformation = new ArrayList<>(); + RemoteJobInformation remoteJobInformation = new RemoteJobInformation("pbsLogfile", + "1-1-tps.0"); + remoteJobInformation.setJobId(9101154); + remoteJobsInformation.add(remoteJobInformation); + remoteJobInformation = new RemoteJobInformation("pbsLogfile", "1-1-tps.1"); + remoteJobInformation.setJobId(9102337); + remoteJobsInformation.add(remoteJobInformation); + remoteJobInformation = new RemoteJobInformation("pbsLogfile", "1-1-tps.2"); + remoteJobInformation.setJobId(6020203); + remoteJobsInformation.add(remoteJobInformation); + return remoteJobsInformation; + } + + private void mockRemoteJobUpdates() { + + // Set up the QueueCommandManager to inform the operations class that + // one of the tasks is complete, one is running, and one is still queued + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9101154", + new String[] { "Exit_status" }, " Exit_status = 0"); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9102337", + new String[] { "Exit_status" }, ""); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 6020203", + new String[] { "Exit_status" }, ""); + + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9101154", + new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, + " " + QueueCommandManager.WALLTIME + " = 10:00:00", + " " + QueueCommandManager.SELECT + " = 2:model=bro"); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9102337", + new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, + " " + QueueCommandManager.WALLTIME + " = 05:00:00", + " " + QueueCommandManager.SELECT + " = 2:model=has"); + QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 6020203", + new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, ""); + } +} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperationsTest.java new file mode 100644 index 0000000..f4aab15 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskDisplayDataOperationsTest.java @@ -0,0 +1,89 @@ +package gov.nasa.ziggy.pipeline.definition.database; + +import java.util.List; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; + +import gov.nasa.ziggy.ZiggyDatabaseRule; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; + +public class PipelineTaskDisplayDataOperationsTest { + + @Rule + public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); + + private PipelineTaskDataOperations pipelineTaskDataOperations; + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations; + private PipelineOperationsTestUtils pipelineOperationsTestUtils; + + @Before + public void setUp() { + pipelineTaskDataOperations = new PipelineTaskDataOperations(); + pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + + pipelineOperationsTestUtils = new PipelineOperationsTestUtils(); + pipelineOperationsTestUtils.setUpSingleModulePipeline(); + } + + @Test + public void testPipelineTaskDisplayDataPipelineTask() { + PipelineTaskDisplayData pipelineTaskDisplayData = pipelineTaskDisplayDataOperations + .pipelineTaskDisplayData(pipelineOperationsTestUtils.getPipelineTasks().get(0)); + pipelineOperationsTestUtils.testPipelineTaskDisplayData(pipelineTaskDisplayData); + } + + @Test + public void testPipelineTaskDisplayDataPipelineInstance() { + List pipelineTaskDisplayDataList = pipelineTaskDisplayDataOperations + .pipelineTaskDisplayData(pipelineOperationsTestUtils.pipelineInstance()); + pipelineOperationsTestUtils.testPipelineTaskDisplayData(pipelineTaskDisplayDataList); + } + + @Test + public void testPipelineTaskDisplayDataPipelineInstanceNode() { + List pipelineTaskDisplayDataList = pipelineTaskDisplayDataOperations + .pipelineTaskDisplayData(pipelineOperationsTestUtils.pipelineInstanceNode()); + pipelineOperationsTestUtils.testPipelineTaskDisplayData(pipelineTaskDisplayDataList); + } + + @Test + public void testPipelineTaskDisplayDataPipelineTasks() { + List pipelineTaskDisplayData = pipelineTaskDisplayDataOperations + .pipelineTaskDisplayData(pipelineOperationsTestUtils.getPipelineTasks()); + pipelineOperationsTestUtils.testPipelineTaskDisplayData(pipelineTaskDisplayData); + } + + @Test + public void testTaskCountsPipelineTask() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, 6, 5, 1); + TaskCounts taskCounts = pipelineTaskDisplayDataOperations.taskCounts(pipelineTask); + TaskCountsTest.testTotalSubtaskCounts(6, 5, 1, taskCounts); + } + + @Test + public void testTaskCountsPipelineInstanceNode() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + PipelineInstanceNode pipelineInstanceNode = pipelineOperationsTestUtils + .pipelineInstanceNode(); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, 6, 5, 1); + TaskCounts taskCounts = pipelineTaskDisplayDataOperations.taskCounts(pipelineInstanceNode); + TaskCountsTest.testTotalSubtaskCounts(6, 5, 1, taskCounts); + } + + @Test + public void testTaskCountsListOfPipelineInstanceNode() { + PipelineTask pipelineTask = pipelineOperationsTestUtils.getPipelineTasks().get(0); + List pipelineInstanceNodes = pipelineOperationsTestUtils + .getPipelineInstanceNodes(); + pipelineTaskDataOperations.updateSubtaskCounts(pipelineTask, 6, 5, 1); + TaskCounts taskCounts = pipelineTaskDisplayDataOperations.taskCounts(pipelineInstanceNodes); + TaskCountsTest.testTotalSubtaskCounts(6, 5, 1, taskCounts); + } +} diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperationsTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperationsTest.java index 0633e05..c770291 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/PipelineTaskOperationsTest.java @@ -8,18 +8,14 @@ import static org.mockito.Mockito.spy; import static org.mockito.Mockito.when; -import java.util.ArrayList; -import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; -import org.hibernate.Hibernate; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; -import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyPropertyRule; @@ -36,13 +32,11 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; -import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; import gov.nasa.ziggy.pipeline.definition.ProcessingStep; -import gov.nasa.ziggy.pipeline.definition.RemoteJob; import gov.nasa.ziggy.pipeline.definition.TaskCounts; -import gov.nasa.ziggy.pipeline.definition.TaskExecutionLog; +import gov.nasa.ziggy.pipeline.definition.TaskCountsTest; import gov.nasa.ziggy.services.database.DatabaseOperations; +import gov.nasa.ziggy.uow.UnitOfWork; import gov.nasa.ziggy.util.ZiggyCollectionUtils; /** @@ -63,8 +57,9 @@ public class PipelineTaskOperationsTest { private PipelineDefinitionNode definitionNode; private PipelineDefinition pipelineDefinition; private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskDataOperations pipelineTaskDataOperations; + private PipelineTaskDisplayDataOperations pipelineTaskDisplayDataOperations; private PipelineInstanceOperations pipelineInstanceOperations; - private PipelineInstanceNodeOperations pipelineInstanceNodeOperations; private QueueCommandManager cmdManager; private TestOperations testOperations = new TestOperations(); @@ -78,11 +73,12 @@ public class PipelineTaskOperationsTest { @Before public void setUp() { - pipelineTaskOperations = Mockito.spy(PipelineTaskOperations.class); - pipelineInstanceOperations = Mockito.spy(PipelineInstanceOperations.class); - pipelineInstanceNodeOperations = Mockito.spy(PipelineInstanceNodeOperations.class); + pipelineTaskOperations = new PipelineTaskOperations(); + pipelineTaskDataOperations = spy(PipelineTaskDataOperations.class); + pipelineTaskDisplayDataOperations = new PipelineTaskDisplayDataOperations(); + pipelineInstanceOperations = spy(PipelineInstanceOperations.class); cmdManager = spy(QueueCommandManager.newInstance()); - when(pipelineTaskOperations.queueCommandManager()).thenReturn(cmdManager); + when(pipelineTaskDataOperations.queueCommandManager()).thenReturn(cmdManager); when(pipelineInstanceOperations.pipelineTaskOperations()) .thenReturn(pipelineTaskOperations); } @@ -124,8 +120,9 @@ public void testPipelineDefinitionNode() { } /** - * Tests that the {@link PipelineTaskOperations#updateProcessingStep(long, ProcessingStep)} - * method performs correctly in the case where all tasks run to completion without errors. In + * Tests that the + * {@link PipelineTaskDataOperations#updateProcessingStep(PipelineTask, ProcessingStep)} method + * performs correctly in the case where all tasks run to completion without errors. In * particular, the pipeline task execution clocks should start and stop at the correct times, * the pipeline instance task execution clock should start and stop at the correct times, and * the pipeline instance and pipeline instance node task counts should be correct. @@ -139,90 +136,84 @@ public void testUpdateProcessingStepNoErrors() { List databaseTasks = testOperations.allPipelineTasks(); for (PipelineTask task : databaseTasks) { - assertTrue(task.getCurrentExecutionStartTimeMillis() <= 0); - assertEquals(ProcessingStep.INITIALIZING, task.getProcessingStep()); + assertFalse(pipelineTaskDataOperations.executionClock(task).isRunning()); + assertEquals(ProcessingStep.WAITING_TO_RUN, + pipelineTaskDataOperations.processingStep(task)); } PipelineInstance instance = testOperations.allPipelineInstances().get(0); - assertTrue(instance.getCurrentExecutionStartTimeMillis() <= 0); + assertFalse(instance.getExecutionClock().isRunning()); assertEquals(PipelineInstance.State.INITIALIZED, instance.getState()); TaskCounts counts = pipelineInstanceOperations.taskCounts(instance); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); - counts = pipelineInstanceNodeOperations.taskCounts(pipelineInstanceNode); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); + counts = pipelineTaskDisplayDataOperations.taskCounts(pipelineInstanceNode); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); // Move the tasks one at a time to the WAITING_TO_RUN step. - PipelineTask updatedTask = pipelineTaskOperations.updateProcessingStep(databaseTasks.get(0), - ProcessingStep.WAITING_TO_RUN); - PipelineInstance pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertEquals(ProcessingStep.WAITING_TO_RUN, updatedTask.getProcessingStep()); - assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); - counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); - assertTrue(updatedTask.getCurrentExecutionStartTimeMillis() > 0); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); - - updatedTask = pipelineTaskOperations.updateProcessingStep(databaseTasks.get(1), - ProcessingStep.WAITING_TO_RUN); - pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertEquals(ProcessingStep.WAITING_TO_RUN, updatedTask.getProcessingStep()); - assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); - counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); - assertTrue(updatedTask.getCurrentExecutionStartTimeMillis() > 0); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + moveTaskToWaitingToRun(databaseTasks.get(0)); + moveTaskToWaitingToRun(databaseTasks.get(1)); // Make sure that the state changes are actually persisted to the database List submittedTasks = testOperations.allPipelineTasks(); for (PipelineTask task : submittedTasks) { - assertEquals(ProcessingStep.WAITING_TO_RUN, task.getProcessingStep()); - assertTrue(task.getCurrentExecutionStartTimeMillis() > 0); + assertEquals(ProcessingStep.WAITING_TO_RUN, + pipelineTaskDataOperations.processingStep(task)); + assertTrue(pipelineTaskDataOperations.executionClock(task).isRunning()); } instance = testOperations.allPipelineInstances().get(0); assertEquals(PipelineInstance.State.PROCESSING, instance.getState()); - assertTrue(instance.getCurrentExecutionStartTimeMillis() > 0); + assertTrue(instance.getExecutionClock().isRunning()); // Move the tasks to the EXECUTING step. - pipelineTaskOperations.updateProcessingStep(submittedTasks.get(0), - ProcessingStep.EXECUTING); - updatedTask = pipelineTaskOperations.updateProcessingStep(submittedTasks.get(1), + pipelineTaskDataOperations.updateProcessingStep(submittedTasks.get(0), ProcessingStep.EXECUTING); + PipelineTask pipelineTask = submittedTasks.get(1); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.EXECUTING); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(pipelineTask.getPipelineInstanceId()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 0, counts); + TaskCountsTest.testTaskCounts(2, 0, 0, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(pipelineTask)); + TaskCountsTest.testTaskCounts(2, 0, 0, 0, counts); // Move both tasks to the COMPLETE step. List tasks = testOperations.allPipelineTasks(); - pipelineTaskOperations.updateProcessingStep(tasks.get(0), ProcessingStep.COMPLETE); - updatedTask = pipelineTaskOperations.updateProcessingStep(tasks.get(1), - ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(tasks.get(0), ProcessingStep.COMPLETE); + pipelineTask = tasks.get(1); + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, ProcessingStep.COMPLETE); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(pipelineTask.getPipelineInstanceId()); assertEquals(PipelineInstance.State.COMPLETED, pipelineInstance.getState()); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() <= 0); + assertFalse(pipelineInstance.getExecutionClock().isRunning()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(pipelineTask)); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); // Make sure the tasks got their clocks stopped. tasks = testOperations.allPipelineTasks(); - assertTrue(tasks.get(0).getCurrentExecutionStartTimeMillis() <= 0); - assertTrue(tasks.get(1).getCurrentExecutionStartTimeMillis() <= 0); + assertFalse(pipelineTaskDataOperations.executionClock(tasks.get(0)).isRunning()); + assertFalse(pipelineTaskDataOperations.executionClock(tasks.get(1)).isRunning()); + } + + private void moveTaskToWaitingToRun(PipelineTask pipelineTask) { + pipelineTaskDataOperations.updateProcessingStep(pipelineTask, + ProcessingStep.WAITING_TO_RUN); + PipelineInstance pipelineInstance = pipelineInstanceOperations + .pipelineInstance(pipelineTask.getPipelineInstanceId()); + assertEquals(ProcessingStep.WAITING_TO_RUN, + pipelineTaskDataOperations.processingStep(pipelineTask)); + assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); + TaskCounts counts = pipelineInstanceOperations.taskCounts(pipelineInstance); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(pipelineTask)); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); + assertTrue(pipelineTaskDataOperations.executionClock(pipelineTask).isRunning()); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); } /** @@ -234,42 +225,42 @@ public void testUpdateProcessingStepWithErrors() { setUpSingleModulePipeline(); // Move the tasks to the EXECUTING step. - pipelineTaskOperations.updateProcessingStep(task1, ProcessingStep.EXECUTING); - pipelineTaskOperations.updateProcessingStep(task2, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task1, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.EXECUTING); // Move one task as errored. - PipelineTask updatedTask = pipelineTaskOperations.taskErrored(task1); + pipelineTaskDataOperations.taskErrored(task1); PipelineInstance pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task1.getPipelineInstanceId()); // Check that the updated states are all correct. assertEquals(PipelineInstance.State.ERRORS_RUNNING, pipelineInstance.getState()); - assertTrue(updatedTask.isError()); + assertTrue(pipelineTaskDataOperations.hasErrored(task1)); TaskCounts counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 1, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 1, counts); + TaskCountsTest.testTaskCounts(2, 0, 0, 1, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task1)); + TaskCountsTest.testTaskCounts(2, 0, 0, 1, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); - assertTrue(updatedTask.getCurrentExecutionStartTimeMillis() <= 0); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); + assertFalse(pipelineTaskDataOperations.executionClock(task1).isRunning()); // Move the other task to the COMPLETE step. - updatedTask = pipelineTaskOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task2.getPipelineInstanceId()); // Check that the updated states are all correct. assertEquals(PipelineInstance.State.ERRORS_STALLED, pipelineInstance.getState()); - assertEquals(ProcessingStep.COMPLETE, updatedTask.getProcessingStep()); + assertEquals(ProcessingStep.COMPLETE, pipelineTaskDataOperations.processingStep(task2)); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 1, 1, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 1, 1, counts); + TaskCountsTest.testTaskCounts(2, 0, 1, 1, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task2)); + TaskCountsTest.testTaskCounts(2, 0, 1, 1, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() <= 0); - assertTrue(updatedTask.getCurrentExecutionStartTimeMillis() <= 0); + assertFalse(pipelineInstance.getExecutionClock().isRunning()); + assertFalse(pipelineTaskDataOperations.executionClock(task2).isRunning()); } /** @@ -296,57 +287,55 @@ public void testMultipleInstanceNodesNoErrors() { // Mark the first 2 tasks as complete and check that all the correct states are // generated. Then create 2 new pipeline tasks in the second instance node. - pipelineTaskOperations.updateProcessingStep(task1, ProcessingStep.COMPLETE); - PipelineTask updatedTask = pipelineTaskOperations.updateProcessingStep(task2, - ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task1, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); PipelineInstance pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task1.getPipelineInstanceId()); assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); TaskCounts counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); - - task3 = new PipelineTask(pipelineInstance, newInstanceNode); - task4 = new PipelineTask(pipelineInstance, newInstanceNode); - testOperations.persistPipelineTasks(List.of(task3, task4)); - newInstanceNode.addPipelineTask(task3); - newInstanceNode.addPipelineTask(task4); - newInstanceNode = testOperations.merge(newInstanceNode); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task1)); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); + + List unitsOfWork = List.of(new UnitOfWork("brief3"), new UnitOfWork("brief4")); + List pipelineTasks = new RuntimeObjectFactory() + .newPipelineTasks(newInstanceNode, pipelineInstance, unitsOfWork); + task3 = pipelineTasks.get(0); + task4 = pipelineTasks.get(1); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 2, 2, 0, counts); + TaskCountsTest.testTaskCounts(4, 2, 2, 0, counts); - counts = pipelineInstanceNodeOperations.taskCounts(newInstanceNode); - PipelineOperationsTestUtils.testTaskCounts(2, 2, 0, 0, counts); + counts = pipelineTaskDisplayDataOperations.taskCounts(newInstanceNode); + TaskCountsTest.testTaskCounts(2, 2, 0, 0, counts); // Move the new tasks to the EXECUTING step. - pipelineTaskOperations.updateProcessingStep(task3, ProcessingStep.EXECUTING); - updatedTask = pipelineTaskOperations.updateProcessingStep(task4, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task3, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task4, ProcessingStep.EXECUTING); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task4.getPipelineInstanceId()); assertEquals(PipelineInstance.State.PROCESSING, pipelineInstance.getState()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 2, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 0, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + TaskCountsTest.testTaskCounts(4, 0, 2, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task4)); + TaskCountsTest.testTaskCounts(2, 0, 0, 0, counts); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); // Move the tasks to COMPLETE. - pipelineTaskOperations.updateProcessingStep(task3, ProcessingStep.COMPLETE); - updatedTask = pipelineTaskOperations.updateProcessingStep(task4, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task3, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task4, ProcessingStep.COMPLETE); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); + .pipelineInstance(task4.getPipelineInstanceId()); assertEquals(PipelineInstance.State.COMPLETED, pipelineInstance.getState()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(4, 0, 4, 0, counts); - counts = pipelineInstanceNodeOperations - .taskCounts(pipelineTaskOperations.pipelineInstanceNode(updatedTask)); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 2, 0, counts); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() <= 0); + TaskCountsTest.testTaskCounts(4, 0, 4, 0, counts); + counts = pipelineTaskDisplayDataOperations + .taskCounts(pipelineTaskOperations.pipelineInstanceNode(task4)); + TaskCountsTest.testTaskCounts(2, 0, 2, 0, counts); + assertFalse(pipelineInstance.getExecutionClock().isRunning()); } @Test @@ -369,181 +358,44 @@ public void testMultipleInstanceNodesWithErrors() { pipelineInstance = testOperations.merge(pipelineInstance); // Move the first task into the EXECUTING step. - PipelineTask updatedTask = pipelineTaskOperations.updateProcessingStep(task1, - ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task1, ProcessingStep.EXECUTING); PipelineInstance pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + .pipelineInstance(task1.getPipelineInstanceId()); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); // Now mark the first task as errored. - updatedTask = pipelineTaskOperations.taskErrored(task1); + pipelineTaskDataOperations.taskErrored(task1); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + .pipelineInstance(task1.getPipelineInstanceId()); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); assertEquals(PipelineInstance.State.ERRORS_RUNNING, pipelineInstance.getState()); TaskCounts counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 1, 0, 1, counts); + TaskCountsTest.testTaskCounts(2, 1, 0, 1, counts); // Move the second task into EXECUTING. - updatedTask = pipelineTaskOperations.updateProcessingStep(task2, ProcessingStep.EXECUTING); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.EXECUTING); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() > 0); + .pipelineInstance(task2.getPipelineInstanceId()); + assertTrue(pipelineInstance.getExecutionClock().isRunning()); // When the second task completes, the instance should go to ERRORS_STALLED even // though there are pipeline instance nodes that have not yet been run. - updatedTask = pipelineTaskOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); + pipelineTaskDataOperations.updateProcessingStep(task2, ProcessingStep.COMPLETE); pipelineInstance = pipelineInstanceOperations - .pipelineInstance(updatedTask.getPipelineInstanceId()); - assertTrue(pipelineInstance.getCurrentExecutionStartTimeMillis() <= 0); + .pipelineInstance(task2.getPipelineInstanceId()); + assertFalse(pipelineInstance.getExecutionClock().isRunning()); assertEquals(PipelineInstance.State.ERRORS_STALLED, pipelineInstance.getState()); counts = pipelineInstanceOperations.taskCounts(pipelineInstance); - PipelineOperationsTestUtils.testTaskCounts(2, 0, 1, 1, counts); - } - - @Test - public void testCreateRemoteJobsFromQstat() { - - PipelineTask task = createPipelineTask(); - pipelineTaskOperations.createRemoteJobsFromQstat(task.getId()); - - // retrieve the task from the database - Set remoteJobs = testOperations.remoteJobs(1L); - assertEquals(3, remoteJobs.size()); - assertTrue(remoteJobs.contains(new RemoteJob(9101154))); - assertTrue(remoteJobs.contains(new RemoteJob(9102337))); - assertTrue(remoteJobs.contains(new RemoteJob(6020203))); - for (RemoteJob job : remoteJobs) { - assertFalse(job.isFinished()); - assertEquals(0, job.getCostEstimate(), 1e-9); - } - } - - @Test - public void testUpdateJobs() { - - // Create the task with 3 remote jobs - createPipelineTask(); - pipelineTaskOperations.createRemoteJobsFromQstat(1L); - - mockRemoteJobUpdates(); - - PipelineTask task = testOperations.updateJobs(1L); - - // Check for the expected results - List remoteJobs = new ArrayList<>(pipelineTaskOperations.remoteJobs(task)); - RemoteJob job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9101154))); - assertTrue(job.isFinished()); - assertEquals(20.0, job.getCostEstimate(), 1e-9); - job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9102337))); - assertFalse(job.isFinished()); - assertEquals(8.0, job.getCostEstimate(), 1e-9); - job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(6020203))); - assertFalse(job.isFinished()); - assertEquals(0, job.getCostEstimate(), 1e-9); - - // Make sure that the database was also updated - Set databaseRemoteJobs = testOperations.remoteJobs(1L); - for (RemoteJob remoteJob : databaseRemoteJobs) { - RemoteJob otherJob = remoteJobs.get(remoteJobs.indexOf(remoteJob)); - assertEquals(otherJob.isFinished(), remoteJob.isFinished()); - assertEquals(otherJob.getCostEstimate(), remoteJob.getCostEstimate(), 1e-9); - } - } - - @Test - public void testUpdateJobsForPipelineInstance() { - - // Create the task with 3 remote jobs - PipelineTask task = createPipelineTask(); - pipelineTaskOperations.createRemoteJobsFromQstat(1L); - - mockRemoteJobUpdates(); - List tasks = testOperations - .updateJobs(pipelineTaskOperations.pipelineInstance(task)); - - assertEquals(1, tasks.size()); - PipelineTask updatedTask = tasks.get(0); - - // Check for the expected results - List remoteJobs = new ArrayList<>(updatedTask.getRemoteJobs()); - RemoteJob job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9101154))); - assertTrue(job.isFinished()); - assertEquals(20.0, job.getCostEstimate(), 1e-9); - job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(9102337))); - assertFalse(job.isFinished()); - assertEquals(8.0, job.getCostEstimate(), 1e-9); - job = remoteJobs.get(remoteJobs.indexOf(new RemoteJob(6020203))); - assertFalse(job.isFinished()); - assertEquals(0, job.getCostEstimate(), 1e-9); - - // Make sure that the database was also updated - PipelineTask databaseTask = testOperations.initializeRemoteJobs(1L); - assertEquals(3, databaseTask.getRemoteJobs().size()); - for (RemoteJob remoteJob : databaseTask.getRemoteJobs()) { - RemoteJob otherJob = remoteJobs.get(remoteJobs.indexOf(remoteJob)); - assertEquals(otherJob.isFinished(), remoteJob.isFinished()); - assertEquals(otherJob.getCostEstimate(), remoteJob.getCostEstimate(), 1e-9); - } + TaskCountsTest.testTaskCounts(2, 0, 1, 1, counts); } @Test public void testTaskIdsForPipelineDefinitionNode() { - setUpSingleModulePipeline(); - List taskIds = pipelineTaskOperations.taskIdsForPipelineDefinitionNode(task1); - assertTrue(taskIds.contains(1L)); - assertTrue(taskIds.contains(2L)); + List taskIds = pipelineTaskOperations.tasksForPipelineDefinitionNode(task1); assertEquals(2, taskIds.size()); - } - - @Test - public void testSummaryMetrics() { - PipelineTask pipelineTask = createPipelineTask(); - List summaryMetrics = new ArrayList<>(); - summaryMetrics.add(new PipelineTaskMetrics("dummy1", 100, Units.TIME)); - summaryMetrics.add(new PipelineTaskMetrics("dummy2", 200, Units.BYTES)); - pipelineTask.getSummaryMetrics().addAll(summaryMetrics); - pipelineTaskOperations.merge(pipelineTask); - pipelineTask.getSummaryMetrics().clear(); - List databaseMetrics = pipelineTaskOperations - .summaryMetrics(pipelineTask); - Map metricsByCategory = new HashMap<>(); - for (PipelineTaskMetrics metric : databaseMetrics) { - metricsByCategory.put(metric.getCategory(), metric); - } - PipelineTaskMetrics dummy1Metric = metricsByCategory.get("dummy1"); - assertNotNull(dummy1Metric); - assertEquals(100L, dummy1Metric.getValue()); - assertEquals(Units.TIME, dummy1Metric.getUnits()); - PipelineTaskMetrics dummy2Metric = metricsByCategory.get("dummy2"); - assertNotNull(dummy2Metric); - assertEquals(200L, dummy2Metric.getValue()); - assertEquals(Units.BYTES, dummy2Metric.getUnits()); - assertEquals(2, metricsByCategory.size()); - } - - @Test - public void testExecLogs() { - PipelineTask pipelineTask = createPipelineTask(); - List taskExecutionLogs = new ArrayList<>(); - taskExecutionLogs.add(new TaskExecutionLog("dummy1", 1)); - taskExecutionLogs.add(new TaskExecutionLog("dummy2", 2)); - pipelineTask.getExecLog().addAll(taskExecutionLogs); - pipelineTaskOperations.merge(pipelineTask); - pipelineTask.getExecLog().clear(); - List execLogs = pipelineTaskOperations.execLogs(pipelineTask); - Map logsByWorkerHost = new HashMap<>(); - for (TaskExecutionLog log : execLogs) { - logsByWorkerHost.put(log.getWorkerHost(), log); - } - TaskExecutionLog log1 = logsByWorkerHost.get("dummy1"); - assertNotNull(log1); - assertEquals(1, log1.getWorkerThread()); - TaskExecutionLog log2 = logsByWorkerHost.get("dummy2"); - assertNotNull(log2); - assertEquals(2, log2.getWorkerThread()); - assertEquals(2, logsByWorkerHost.size()); + assertTrue(taskIds.contains(task1)); + assertTrue(taskIds.contains(task2)); } @Test @@ -595,7 +447,7 @@ public void testParameterSets() { assertEquals(4, parameterSetNames.size()); } - private PipelineTask createPipelineTask() { + PipelineTask createPipelineTask() { PipelineInstance instance = testOperations.merge(new PipelineInstance()); PipelineModuleDefinition modDef = testOperations.merge(new PipelineModuleDefinition("tps")); @@ -614,7 +466,7 @@ private PipelineTask createPipelineTask() { defNode.addAllOutputDataFileTypes(List.of(outputDataFileType)); defNode.addAllModelTypes(List.of(modelType)); defNode = testOperations.merge(defNode); - PipelineTask task = new PipelineTask(instance, instNode); + PipelineTask task = new PipelineTask(instance, instNode, null); task = pipelineTaskOperations.merge(task); instNode.addPipelineTask(task); instNode = testOperations.merge(instNode); @@ -638,29 +490,6 @@ private PipelineTask createPipelineTask() { return task; } - private void mockRemoteJobUpdates() { - - // Set up the QueueCommandManager to inform the operations class that - // one of the tasks is complete, one is running, and one is still queued - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9101154", - new String[] { "Exit_status" }, " Exit_status = 0"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9102337", - new String[] { "Exit_status" }, ""); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 6020203", - new String[] { "Exit_status" }, ""); - - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9101154", - new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, - " " + QueueCommandManager.WALLTIME + " = 10:00:00", - " " + QueueCommandManager.SELECT + " = 2:model=bro"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 9102337", - new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, - " " + QueueCommandManager.WALLTIME + " = 05:00:00", - " " + QueueCommandManager.SELECT + " = 2:model=has"); - QueueCommandManagerTest.mockQstatCall(cmdManager, "-xf 6020203", - new String[] { QueueCommandManager.SELECT, QueueCommandManager.WALLTIME }, ""); - } - private void persistParameterSets() { ParameterSet parameterSet = new ParameterSet("parameter1"); parameterSet.getParameters() @@ -709,10 +538,6 @@ public PipelineInstance merge(PipelineInstance pipelineInstance) { return performTransaction(() -> new PipelineInstanceCrud().merge(pipelineInstance)); } - public void persistPipelineTasks(List pipelineTasks) { - performTransaction(() -> new PipelineTaskCrud().persist(pipelineTasks)); - } - public PipelineDefinition merge(PipelineDefinition pipelineDefinition) { return performTransaction(() -> new PipelineDefinitionCrud().merge(pipelineDefinition)); } @@ -724,41 +549,6 @@ public DataFileType merge(DataFileType dataFileType) { }); } - public Set remoteJobs(long taskId) { - return performTransaction(() -> { - PipelineTask task = new PipelineTaskCrud().retrieve(taskId); - Hibernate.initialize(task.getRemoteJobs()); - return task.getRemoteJobs(); - }); - } - - public PipelineTask updateJobs(long taskId) { - return performTransaction(() -> { - PipelineTask pipelineTask = new PipelineTaskCrud().retrieve(taskId); - pipelineTaskOperations.updateJobs(pipelineTask); - return pipelineTask; - }); - } - - public List updateJobs(PipelineInstance pipelineInstance) { - return performTransaction(() -> { - List pipelineTasks = pipelineInstanceOperations - .updateJobs(pipelineInstance); - for (PipelineTask pipelineTask : pipelineTasks) { - Hibernate.initialize(pipelineTask.getRemoteJobs()); - } - return pipelineTasks; - }); - } - - public PipelineTask initializeRemoteJobs(long taskId) { - return performTransaction(() -> { - PipelineTask pipelineTask = new PipelineTaskCrud().retrieve(taskId); - Hibernate.initialize(pipelineTask.getRemoteJobs()); - return pipelineTask; - }); - } - public void persist(ParameterSet parameterSet) { performTransaction(() -> new ParameterSetCrud().persist(parameterSet)); } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactoryTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactoryTest.java index 73b2864..ec65f65 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactoryTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/database/RuntimeObjectFactoryTest.java @@ -11,7 +11,7 @@ import java.util.Map; import java.util.Set; -import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.collections4.CollectionUtils; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -190,23 +190,25 @@ public void testNewPipelineTasks() { assertNotNull(pipelineTask); assertEquals(pipelineInstance.getId().longValue(), pipelineTask.getPipelineInstanceId()); assertEquals(definitionNode.getModuleName(), pipelineTask.getModuleName()); - UnitOfWork taskUow = pipelineTask.uowTaskInstance(); + UnitOfWork taskUow = pipelineTask.getUnitOfWork(); Set parameters = taskUow.getParameters(); assertFalse(CollectionUtils.isEmpty(parameters)); Parameter parameter = parameters.iterator().next(); assertEquals("bauhaus", parameter.getName()); assertEquals(1, parameters.size()); + assertEquals(pipelineTask.getUnitOfWork().getParameters(), taskUow.getParameters()); pipelineTask = pipelineTasksById.get(4L); assertNotNull(pipelineTask); assertEquals(pipelineInstance.getId().longValue(), pipelineTask.getPipelineInstanceId()); assertEquals(definitionNode.getModuleName(), pipelineTask.getModuleName()); - taskUow = pipelineTask.uowTaskInstance(); + taskUow = pipelineTask.getUnitOfWork(); parameters = taskUow.getParameters(); assertFalse(CollectionUtils.isEmpty(parameters)); parameter = parameters.iterator().next(); assertEquals("duran", parameter.getName()); assertEquals(1, parameters.size()); + assertEquals(pipelineTask.getUnitOfWork().getParameters(), taskUow.getParameters()); // In addition to the 2 tasks created here, there were already 2 in the // database from PipelineOperationsTestUtils. diff --git a/src/test/java/gov/nasa/ziggy/services/alert/AlertLogCrudTest.java b/src/test/java/gov/nasa/ziggy/services/alert/AlertLogCrudTest.java index f31a04e..6c85d86 100644 --- a/src/test/java/gov/nasa/ziggy/services/alert/AlertLogCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/services/alert/AlertLogCrudTest.java @@ -1,16 +1,15 @@ package gov.nasa.ziggy.services.alert; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import java.text.SimpleDateFormat; -import java.util.ArrayList; import java.util.Date; import java.util.List; import org.junit.Before; import org.junit.Rule; import org.junit.Test; -import org.slf4j.event.Level; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.module.PipelineException; @@ -18,6 +17,7 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineInstanceCrud; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskCrud; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.database.DatabaseOperations; // TODO Rename to AlertLogOperationsTest and adjust @@ -35,12 +35,20 @@ public class AlertLogCrudTest { private Date date1; private Date date2; - private Date date3; private Date date4; private Date date5; private Date date6; private Date date7; + private PipelineTask task1; + private PipelineTask task3; + private PipelineTask task4; + private PipelineTask task5; + private PipelineInstance instance1; + private PipelineInstance instance2; + + private AlertLogOperations alertLogOperations = new AlertLogOperations(); + @Rule public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); @@ -49,7 +57,6 @@ public void setUp() throws Exception { date1 = parser.parse("Jun-1-12 12:00:00"); date2 = parser.parse("Jun-2-12 12:00:00"); - date3 = parser.parse("Jul-10-12 15:00:00"); date4 = parser.parse("Aug-12-12 02:00:00"); date5 = parser.parse("Sep-20-12 05:00:00"); date6 = parser.parse("Sep-21-12 05:00:00"); @@ -66,27 +73,28 @@ public void testRetrieveComponents() { components = testOperations.alertComponents(); // Check number of components as well as sort. - assertEquals(8, components.size()); + assertEquals(6, components.size()); assertEquals("s1", components.get(0)); assertEquals("s2", components.get(1)); - assertEquals("s3", components.get(2)); - assertEquals("s4", components.get(3)); + assertEquals("s4", components.get(2)); + assertEquals("s5", components.get(3)); + assertEquals("s6", components.get(4)); + assertEquals("s8", components.get(5)); } @Test public void testRetrieveSeverities() { - List severities = testOperations.alertSeverities(); + List severities = testOperations.alertSeverities(); assertEquals(0, severities.size()); populateObjects(); severities = testOperations.alertSeverities(); // Check number of components as well as sort. - assertEquals(4, severities.size()); - assertEquals(Level.DEBUG.toString(), severities.get(0)); - assertEquals(Level.ERROR.toString(), severities.get(1)); - assertEquals(Level.INFO.toString(), severities.get(2)); - assertEquals(Level.WARN.toString(), severities.get(3)); + assertEquals(3, severities.size()); + assertEquals(Severity.ERROR, severities.get(0)); + assertEquals(Severity.INFRASTRUCTURE, severities.get(1)); + assertEquals(Severity.WARNING, severities.get(2)); } @Test @@ -94,7 +102,7 @@ public void testCreateRetrieve() throws Exception { populateObjects(); List alerts = testOperations.alertsInDateRange(date2, date6); - assertEquals("alerts.size()", 6, alerts.size()); + assertEquals("alerts.size()", 4, alerts.size()); } @Test(expected = PipelineException.class) @@ -114,18 +122,13 @@ public void testRetrieveNullComponent() { @Test(expected = PipelineException.class) public void testRetrieveNullSeverity() { - testOperations.alerts(new Date(), new Date(), new String[0], null); + testOperations.alerts(new Date(), new Date(), List.of(), null); } @Test - public void testRetrieveAlertsByTaskId() { + public void testRetrieveAlertsByTask() { populateObjects(); - - List taskIds = new ArrayList<>(); - taskIds.add(5L); - taskIds.add(1L); - - List alerts = testOperations.alertsForPipelineTasks(taskIds); + List alerts = testOperations.alertsForPipelineTasks(List.of(task5, task1)); assertEquals(4, alerts.size()); } @@ -134,101 +137,107 @@ public void testRetrieve() { populateObjects(); // Test that empty components means that all components are considered. - String[] components = {}; - String[] severities = {}; + List components = List.of(); + List severities = List.of(); List alerts = testOperations.alerts(date2, date6, components, severities); - assertEquals(6, alerts.size()); + assertEquals(4, alerts.size()); // This component isn't in the date range. - components = new String[] { "s4" }; + components = List.of("s4"); alerts = testOperations.alerts(date2, date6, components, severities); assertEquals(0, alerts.size()); // This component has one entry in and one outside of the range. - components[0] = "s1"; + components = List.of("s1"); alerts = testOperations.alerts(date2, date6, components, severities); assertEquals(1, alerts.size()); assertEquals("D", alerts.get(0).getAlertData().getProcessName()); // Specify all components; check sort. - components = new String[] { "s1", "s2", "s3", "s4" }; - severities = new String[] { Level.ERROR.toString(), Level.ERROR.toString(), - Level.DEBUG.toString(), Level.INFO.toString(), Level.WARN.toString(), - Level.ERROR.toString() }; + components = List.of("s1", "s2", "s4"); + severities = List.of(Severity.ERROR, Severity.ERROR, Severity.INFRASTRUCTURE, + Severity.WARNING, Severity.ERROR); alerts = testOperations.alerts(date1, date7, components, severities); - assertEquals(5, alerts.size()); + assertEquals(4, alerts.size()); assertEquals("D", alerts.get(0).getAlertData().getProcessName()); assertEquals("E", alerts.get(1).getAlertData().getProcessName()); assertEquals("C", alerts.get(2).getAlertData().getProcessName()); - assertEquals("B", alerts.get(3).getAlertData().getProcessName()); - assertEquals("A", alerts.get(4).getAlertData().getProcessName()); - assertEquals(Level.ERROR.toString(), alerts.get(0).getAlertData().getSeverity()); + assertEquals("A", alerts.get(3).getAlertData().getProcessName()); + assertEquals(Severity.ERROR, alerts.get(0).getAlertData().getSeverity()); - components = new String[] { "s5", "s6", "s7", "s8" }; + components = List.of("s5", "s6", "s8"); alerts = testOperations.alerts(date1, date7, components, severities); - assertEquals(5, alerts.size()); - assertEquals(Level.ERROR.toString(), alerts.get(0).getAlertData().getSeverity()); - assertEquals(Level.ERROR.toString(), alerts.get(1).getAlertData().getSeverity()); - assertEquals(Level.DEBUG.toString(), alerts.get(2).getAlertData().getSeverity()); - assertEquals(Level.INFO.toString(), alerts.get(3).getAlertData().getSeverity()); - assertEquals(Level.WARN.toString(), alerts.get(4).getAlertData().getSeverity()); + assertEquals(4, alerts.size()); + assertEquals(Severity.ERROR, alerts.get(0).getAlertData().getSeverity()); + assertEquals(Severity.ERROR, alerts.get(1).getAlertData().getSeverity()); + assertEquals(Severity.INFRASTRUCTURE, alerts.get(2).getAlertData().getSeverity()); + assertEquals(Severity.WARNING, alerts.get(3).getAlertData().getSeverity()); // Check just the SEVERE severity. - severities = new String[] { Level.ERROR.toString() }; + severities = List.of(Severity.ERROR); alerts = testOperations.alerts(date1, date7, components, severities); assertEquals(2, alerts.size()); - assertEquals(Level.ERROR.toString(), alerts.get(0).getAlertData().getSeverity()); + assertEquals(Severity.ERROR, alerts.get(0).getAlertData().getSeverity()); } @Test public void testRetrieveForPipelineInstance() { - populateInstancesAndTasks(); populateObjects(); - List logs = testOperations.alertsForPipelineInstance(1L); - assertEquals(4, logs.size()); + List logs = alertLogOperations().alertLogs(instance1); + assertEquals(2, logs.size()); + Alert alert = null; + for (AlertLog log : logs) { + if (log.getAlertData().getProcessName().equals("AW")) { + alert = log.getAlertData(); + } + } + assertNotNull(alert); + assertEquals("s8", alert.getSourceComponent()); + assertEquals("AW", alert.getProcessName()); + assertEquals("a", alert.getProcessHost()); + assertEquals(1L, alert.getProcessId()); + assertEquals("message1", alert.getMessage()); + assertEquals(1L, alert.getSourceTask().getId().longValue()); + assertEquals(1L, alert.getSourceTask().getPipelineInstanceId()); + assertEquals(date1, alert.getTimestamp()); } private void populateObjects() { - testOperations - .persistAlert(new AlertLog(new Alert(date7, "s1", 5, "E", "e", 5, "message5"))); - testOperations - .persistAlert(new AlertLog(new Alert(date5, "s1", 4, "D", "d", 4, "message4"))); - testOperations - .persistAlert(new AlertLog(new Alert(date4, "s2", 3, "C", "c", 3, "message3"))); - testOperations - .persistAlert(new AlertLog(new Alert(date3, "s3", 2, "B", "b", 2, "message2"))); - testOperations - .persistAlert(new AlertLog(new Alert(date1, "s4", 1, "A", "a", 1, "message1"))); - - testOperations.persistAlert(new AlertLog( - new Alert(date5, "s5", 4, "DS", "d", 4, Level.ERROR.toString(), "message4"))); - testOperations.persistAlert(new AlertLog( - new Alert(date7, "s5", 5, "EF", "e", 5, Level.ERROR.toString(), "message5"))); - testOperations.persistAlert(new AlertLog( - new Alert(date4, "s6", 3, "CD", "c", 3, Level.DEBUG.toString(), "message3"))); - testOperations.persistAlert(new AlertLog( - new Alert(date3, "s7", 2, "BI", "b", 2, Level.INFO.toString(), "message2"))); - testOperations.persistAlert(new AlertLog( - new Alert(date1, "s8", 1, "AW", "a", 1, Level.WARN.toString(), "message1"))); - } - - private void populateInstancesAndTasks() { + instance1 = testOperations.merge(new PipelineInstance()); + task1 = new PipelineTask(instance1, null, null); - PipelineInstance instance1 = testOperations.merge(new PipelineInstance()); - PipelineTask task1 = new PipelineTask(instance1, null); - PipelineTask task2 = new PipelineTask(instance1, null); - - PipelineInstance instance2 = testOperations.merge(new PipelineInstance()); - PipelineTask task3 = new PipelineTask(instance2, null); - PipelineTask task4 = new PipelineTask(instance2, null); - PipelineTask task5 = new PipelineTask(instance2, null); + instance2 = testOperations.merge(new PipelineInstance()); + task3 = new PipelineTask(instance2, null, null); + task4 = new PipelineTask(instance2, null, null); + task5 = new PipelineTask(instance2, null, null); testOperations.persistPipelineTask(task1); - testOperations.persistPipelineTask(task2); testOperations.persistPipelineTask(task3); testOperations.persistPipelineTask(task4); testOperations.persistPipelineTask(task5); + + alertLogOperations().persist( + new AlertLog(new Alert(date7, "s1", task5, "E", "e", 5, Severity.ERROR, "message5"))); + alertLogOperations().persist( + new AlertLog(new Alert(date5, "s1", task4, "D", "d", 4, Severity.ERROR, "message4"))); + alertLogOperations().persist(new AlertLog( + new Alert(date4, "s2", task3, "C", "c", 3, Severity.INFRASTRUCTURE, "message3"))); + alertLogOperations().persist( + new AlertLog(new Alert(date1, "s4", task1, "A", "a", 1, Severity.WARNING, "message1"))); + + alertLogOperations().persist( + new AlertLog(new Alert(date5, "s5", task4, "DS", "d", 4, Severity.ERROR, "message4"))); + alertLogOperations().persist( + new AlertLog(new Alert(date7, "s5", task5, "EF", "e", 5, Severity.ERROR, "message5"))); + alertLogOperations().persist(new AlertLog( + new Alert(date4, "s6", task3, "CD", "c", 3, Severity.INFRASTRUCTURE, "message3"))); + alertLogOperations().persist(new AlertLog( + new Alert(date1, "s8", task1, "AW", "a", 1, Severity.WARNING, "message1"))); + } + + AlertLogOperations alertLogOperations() { + return alertLogOperations; } private static class TestOperations extends DatabaseOperations { @@ -237,7 +246,7 @@ public List alertComponents() { return performTransaction(() -> new AlertLogCrud().retrieveComponents()); } - public List alertSeverities() { + public List alertSeverities() { return performTransaction(() -> new AlertLogCrud().retrieveSeverities()); } @@ -245,23 +254,14 @@ public List alertsInDateRange(Date startDate, Date endDate) { return performTransaction(() -> new AlertLogCrud().retrieve(startDate, endDate)); } - public List alerts(Date startDate, Date endDate, String[] components, - String[] severities) { + public List alerts(Date startDate, Date endDate, List components, + List severities) { return performTransaction( () -> new AlertLogCrud().retrieve(startDate, endDate, components, severities)); } - public List alertsForPipelineTasks(List taskIds) { - return performTransaction(() -> new AlertLogCrud().retrieveByPipelineTaskIds(taskIds)); - } - - public List alertsForPipelineInstance(long instanceId) { - return performTransaction( - () -> new AlertLogCrud().retrieveForPipelineInstance(instanceId)); - } - - public void persistAlert(AlertLog alertLog) { - performTransaction(() -> new AlertLogCrud().persist(alertLog)); + public List alertsForPipelineTasks(List tasks) { + return performTransaction(() -> new AlertLogCrud().retrieveByPipelineTasks(tasks)); } public PipelineInstance merge(PipelineInstance pipelineInstance) { diff --git a/src/test/java/gov/nasa/ziggy/services/config/KeyValuePairOperationsTest.java b/src/test/java/gov/nasa/ziggy/services/config/KeyValuePairOperationsTest.java deleted file mode 100644 index 9cb127c..0000000 --- a/src/test/java/gov/nasa/ziggy/services/config/KeyValuePairOperationsTest.java +++ /dev/null @@ -1,50 +0,0 @@ -package gov.nasa.ziggy.services.config; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - -import java.util.List; - -import org.junit.Rule; -import org.junit.Test; - -import gov.nasa.ziggy.ZiggyDatabaseRule; - -public class KeyValuePairOperationsTest { - private KeyValuePairOperations keyValuePairOperations = new KeyValuePairOperations(); - - @Rule - public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); - - @Test - public void testPersist() throws Exception { - keyValuePairOperations.persist(new KeyValuePair("key1", "value1")); - - assertEquals("value1", keyValuePairOperations.keyValuePairValue("key1")); - } - - @Test - public void testUpdate() throws Exception { - keyValuePairOperations.persist(new KeyValuePair("key1", "value1")); - assertEquals("value1", keyValuePairOperations.keyValuePairValue("key1")); - - keyValuePairOperations.updateKeyValuePair("key1", "value2"); - assertEquals("value2", keyValuePairOperations.keyValuePairValue("key1")); - } - - @Test - public void testkeyValuePairs() throws Exception { - KeyValuePair keyValuePair1 = new KeyValuePair("key1", "value1"); - keyValuePairOperations.persist(keyValuePair1); - KeyValuePair keyValuePair2 = new KeyValuePair("key2", "value2"); - keyValuePairOperations.persist(keyValuePair2); - KeyValuePair keyValuePair3 = new KeyValuePair("key3", "value3"); - keyValuePairOperations.persist(keyValuePair3); - - List keyValuePairs = keyValuePairOperations.keyValuePairs(); - assertEquals(3, keyValuePairs.size()); - assertTrue(keyValuePairs.contains(keyValuePair1)); - assertTrue(keyValuePairs.contains(keyValuePair2)); - assertTrue(keyValuePairs.contains(keyValuePair3)); - } -} diff --git a/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java b/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java index 0a2b653..ad1e63d 100644 --- a/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java +++ b/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java @@ -200,12 +200,6 @@ public void testConcurrentAccess() throws InterruptedException { } } - @Test - public void testBuildProperty() { - assertNotNull( - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); - } - @Test public void testLogJvmProperties() { // Just ensure that the code is covered and doesn't blow up. If the log output can be easily diff --git a/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java index 4e0ea91..d9bc007 100644 --- a/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java +++ b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java @@ -4,6 +4,7 @@ import static gov.nasa.ziggy.XmlUtils.complexTypeContent; import static gov.nasa.ziggy.ZiggyUnitTestUtils.TEST_DATA; import static gov.nasa.ziggy.services.config.PropertyName.DATA_RECEIPT_DIR; +import static gov.nasa.ziggy.services.config.PropertyName.PIPELINE_HOME_DIR; import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -89,6 +90,10 @@ public class ZiggyEventHandlerTest { public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, DirectoryProperties.ziggyCodeBuildDir().toString()); + @Rule + public ZiggyPropertyRule pipelineHomeDirPropertyRule = new ZiggyPropertyRule(PIPELINE_HOME_DIR, + DirectoryProperties.ziggyCodeBuildDir().toString()); + public ZiggyPropertyRule dataReceiptDirPropertyRule = new ZiggyPropertyRule(DATA_RECEIPT_DIR, directoryRule, TEST_DATA_DIR); @@ -219,7 +224,7 @@ public void testStartPipeline() throws IOException, InterruptedException { assertEquals(1, tasks.size()); PipelineTask task = tasks.get(0); - UnitOfWork uow = task.uowTaskInstance(); + UnitOfWork uow = task.getUnitOfWork(); assertEquals( DirectoryProperties.dataReceiptDir().toAbsolutePath().resolve("gazelle").toString(), DirectoryUnitOfWorkGenerator.directory(uow)); @@ -337,7 +342,7 @@ public void testEventWithTwoReadyFiles() throws IOException, InterruptedExceptio assertEquals(2, tasks.size()); List uowStrings = new ArrayList<>(); for (PipelineTask task : tasks) { - UnitOfWork uow = task.uowTaskInstance(); + UnitOfWork uow = task.getUnitOfWork(); uowStrings.add(DirectoryUnitOfWorkGenerator.directory(uow)); } Path dataReceiptDir = DirectoryProperties.dataReceiptDir().toAbsolutePath(); @@ -451,7 +456,7 @@ public void testNullEventLabel() throws IOException, InterruptedException { assertEquals(1, tasks.size()); PipelineTask task = tasks.get(0); - UnitOfWork uow = task.uowTaskInstance(); + UnitOfWork uow = task.getUnitOfWork(); assertEquals(directoryRule.directory().toAbsolutePath().resolve("events").toString(), DirectoryUnitOfWorkGenerator.directory(uow)); } diff --git a/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventTest.java b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventTest.java new file mode 100644 index 0000000..f631fb5 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventTest.java @@ -0,0 +1,90 @@ +package gov.nasa.ziggy.services.events; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.util.Date; +import java.util.Set; +import java.util.TreeSet; + +import org.junit.Before; +import org.junit.Test; + +import gov.nasa.ziggy.util.SystemProxy; + +/** + * Performs unit tests for {@link ZiggyEvent}. + * + * @author Bill Wohler + */ +public class ZiggyEventTest { + + private static final String EVENT_HANDLER_NAME = "eventHandlerName"; + private static final String PIPELINE_NAME = "pipelineName"; + private static final int PIPELINE_INSTANCE = 1; + private static final Set EVENT_LABELS = new TreeSet<>(Set.of("label1", "label2")); + private static final long TIME1 = 1000; + private ZiggyEvent ziggyEvent; + + @Before + public void setUp() { + SystemProxy.setUserTime(TIME1); + ziggyEvent = new ZiggyEvent(EVENT_HANDLER_NAME, PIPELINE_NAME, PIPELINE_INSTANCE, + EVENT_LABELS); + } + + @Test + public void testGetId() { + assertNull(ziggyEvent.getId()); + } + + @Test + public void testGetEventHandlerName() { + assertEquals(EVENT_HANDLER_NAME, ziggyEvent.getEventHandlerName()); + } + + @Test + public void testGetPipelineName() { + assertEquals(PIPELINE_NAME, ziggyEvent.getPipelineName()); + } + + @Test + public void testGetEventTime() { + assertEquals(new Date(TIME1), ziggyEvent.getEventTime()); + } + + @Test + public void testGetPipelineInstanceId() { + assertEquals(PIPELINE_INSTANCE, ziggyEvent.getPipelineInstanceId()); + } + + @Test + public void testGetEventLabels() { + assertEquals(EVENT_LABELS, ziggyEvent.getEventLabels()); + } + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + ZiggyEvent ziggyEvent1 = ziggyEvent; + SystemProxy.setUserTime(TIME1); + ZiggyEvent ziggyEvent2 = new ZiggyEvent(EVENT_HANDLER_NAME, PIPELINE_NAME, + PIPELINE_INSTANCE, EVENT_LABELS); + + assertEquals(ziggyEvent1.hashCode(), ziggyEvent2.hashCode()); + assertTrue(ziggyEvent1.equals(ziggyEvent1)); + assertTrue(ziggyEvent1.equals(ziggyEvent2)); + + assertFalse(ziggyEvent1.equals(null)); + assertFalse(ziggyEvent1.equals("a string")); + } + + @Test + public void testToString() { + assertEquals("eventHandlerName=" + EVENT_HANDLER_NAME + ", pipelineName=" + PIPELINE_NAME + + ", eventTime=" + new Date(TIME1) + ", pipelineInstanceId=" + PIPELINE_INSTANCE + + ", eventLabels=" + EVENT_LABELS, ziggyEvent.toString()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/services/logging/TaskLogTest.java b/src/test/java/gov/nasa/ziggy/services/logging/TaskLogTest.java index e219d4e..9be300c 100644 --- a/src/test/java/gov/nasa/ziggy/services/logging/TaskLogTest.java +++ b/src/test/java/gov/nasa/ziggy/services/logging/TaskLogTest.java @@ -3,6 +3,8 @@ import static gov.nasa.ziggy.services.config.PropertyName.RESULTS_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; import java.io.File; import java.io.IOException; @@ -21,6 +23,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.core.LoggerContext; import org.junit.After; +import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; @@ -32,6 +35,7 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; @@ -61,8 +65,8 @@ public class TaskLogTest { private static final int NEXT_JOB_INDEX = 11; private File algorithmLog1; - private Long timestamp; + private PipelineTaskDataOperations pipelineTaskDataOperations; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -77,6 +81,12 @@ public class TaskLogTest { public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) .around(resultsDirPropertyRule); + @Before + public void setUp() { + pipelineTaskDataOperations = spy(PipelineTaskDataOperations.class); + TaskLog.setPipelineTaskDataOperations(pipelineTaskDataOperations); + } + @After public void teardown() throws InterruptedException, URISyntaxException, IOException { LoggerContext context = (LoggerContext) LogManager.getContext(false); @@ -86,8 +96,9 @@ public void teardown() throws InterruptedException, URISyntaxException, IOExcept @Test public void testTaskLog() throws IOException { createAndPopulateZiggyTaskLog(INSTANCE_ID, TASK_ID, STEP_INDEX_0, TEST_LOG_MESSAGE_1); + PipelineTask pipelineTask = createPipelineTask(INSTANCE_ID, TASK_ID); File expectedTaskLogFile1 = DirectoryProperties.taskLogDir() - .resolve(createPipelineTask(INSTANCE_ID, TASK_ID, STEP_INDEX_0).logFilename(0)) + .resolve(pipelineTaskDataOperations.logFilename(pipelineTask, 0, STEP_INDEX_0)) .toFile(); assertTrue("log file exists", expectedTaskLogFile1.exists()); @@ -100,8 +111,9 @@ public void testTaskLog() throws IOException { @Test public void testTaskLogEnum() { createAndPopulateZiggyTaskLog(INSTANCE_ID, TASK_ID, STEP_INDEX_0, TEST_LOG_MESSAGE_1); + PipelineTask pipelineTask = createPipelineTask(INSTANCE_ID, TASK_ID); File expectedTaskLogFile1 = DirectoryProperties.taskLogDir() - .resolve(createPipelineTask(INSTANCE_ID, TASK_ID, STEP_INDEX_0).logFilename(0)) + .resolve(pipelineTaskDataOperations.logFilename(pipelineTask, 0, STEP_INDEX_0)) .toFile(); Matcher matcher = TaskLogInformation.LOG_FILE_NAME_PATTERN .matcher(expectedTaskLogFile1.getName()); @@ -113,8 +125,9 @@ public void testTaskLogEnum() { @Test public void testAlgorithmLogEnum() { - PipelineTask task = createPipelineTask(INSTANCE_ID, TASK_ID, STEP_INDEX_4); - String algorithmLogFilename = task.logFilename(JOB_INDEX); + PipelineTask task = createPipelineTask(INSTANCE_ID, TASK_ID); + String algorithmLogFilename = pipelineTaskDataOperations.logFilename(task, JOB_INDEX, + STEP_INDEX_4); Matcher matcher = TaskLogInformation.LOG_FILE_NAME_PATTERN.matcher(algorithmLogFilename); assertTrue(matcher.matches()); @@ -144,7 +157,9 @@ public void testSearchForLogFiles() throws IOException { "extremely_long_test_message"); // Try to get TaskLogInformation for each of the created files - Set taskLogInformationSet = TaskLog.searchForLogFiles(TASK_ID); + PipelineTask pipelineTask = spy(PipelineTask.class); + doReturn(TASK_ID).when(pipelineTask).getId(); + Set taskLogInformationSet = TaskLog.searchForLogFiles(pipelineTask); // Check values in the TaskLogInformation instances. assertEquals(4, taskLogInformationSet.size()); @@ -160,8 +175,9 @@ public void testAlgorithmTaskLog() throws IOException { // Create a log file for the algorithm createAndPopulateAlgorithmTaskLog(INSTANCE_ID, TASK_ID, STEP_INDEX_1, TEST_LOG_MESSAGE_1); + PipelineTask pipelineTask = createPipelineTask(INSTANCE_ID, TASK_ID); algorithmLog1 = DirectoryProperties.algorithmLogsDir() - .resolve(createPipelineTask(INSTANCE_ID, TASK_ID, STEP_INDEX_1).logFilename(0)) + .resolve(pipelineTaskDataOperations.logFilename(pipelineTask, 0, STEP_INDEX_1)) .toFile(); assertTrue("log file exists", algorithmLog1.exists()); @@ -219,6 +235,14 @@ private void checkTaskLogInformationValues(TaskLogInformation taskLogInfo, private void createAndPopulateZiggyTaskLog(long instanceId, long taskId, int stepIndex, String message) { + + PipelineTask pipelineTask = createPipelineTask(instanceId, taskId); + Mockito + .doReturn(pipelineTaskDataOperations.logFilename(pipelineTask, + TaskLog.LOCAL_LOG_FILE_JOB_INDEX, stepIndex)) + .when(pipelineTaskDataOperations) + .logFilename(pipelineTask, TaskLog.LOCAL_LOG_FILE_JOB_INDEX); + CommandLine commandLine = new CommandLine(DirectoryProperties.ziggyHomeDir() .getParent() .resolve("src") @@ -227,8 +251,7 @@ private void createAndPopulateZiggyTaskLog(long instanceId, long taskId, int ste .resolve("ziggy.pl") .toString()); commandLine.addArgument("--verbose"); - commandLine.addArgument( - TaskLog.ziggyLogFileSystemProperty(createPipelineTask(instanceId, taskId, stepIndex))); + commandLine.addArgument(TaskLog.ziggyLogFileSystemProperty(pipelineTask)); commandLine.addArgument("-D" + PropertyName.LOG4J2_CONFIGURATION_FILE.property() + "=" + LOG4J_CONFIG_PATH.toAbsolutePath().toString()); commandLine.addArgument("--class=" + TaskLogCreator.class.getName()); @@ -252,6 +275,13 @@ private void createAndPopulateZiggyTaskLog(long instanceId, long taskId, int ste private void createAndPopulateAlgorithmTaskLog(long instanceId, long taskId, int stepIndex, String message) { + PipelineTask pipelineTask = createPipelineTask(instanceId, taskId); + Mockito + .doReturn(pipelineTaskDataOperations.logFilename(pipelineTask, + TaskLog.LOCAL_LOG_FILE_JOB_INDEX, stepIndex)) + .when(pipelineTaskDataOperations) + .logFilename(pipelineTask, TaskLog.LOCAL_LOG_FILE_JOB_INDEX); + CommandLine commandLine = new CommandLine(DirectoryProperties.ziggyHomeDir() .getParent() .resolve("src") @@ -259,8 +289,7 @@ private void createAndPopulateAlgorithmTaskLog(long instanceId, long taskId, int .resolve("perl") .resolve("ziggy.pl") .toString()); - commandLine.addArgument(TaskLog - .algorithmLogFileSystemProperty(createPipelineTask(instanceId, taskId, stepIndex))); + commandLine.addArgument(TaskLog.algorithmLogFileSystemProperty(pipelineTask)); commandLine.addArgument("-D" + PropertyName.LOG4J2_CONFIGURATION_FILE.property() + "=" + LOG4J_CONFIG_PATH.toAbsolutePath().toString()); commandLine.addArgument("--class=" + TaskLogCreator.class.getName()); @@ -281,21 +310,21 @@ private void createAndPopulateAlgorithmTaskLog(long instanceId, long taskId, int externalProcess.execute(); } - private PipelineTask createPipelineTask(long instanceId, long taskId, int stepIndex) { + private PipelineTask createPipelineTask(long instanceId, long taskId) { PipelineInstance instance = new PipelineInstance(); instance.setId(instanceId); PipelineModuleDefinition module = new PipelineModuleDefinition("testexename"); - PipelineTask task = Mockito - .spy(new PipelineTask(instance, new PipelineInstanceNode(null, module))); + PipelineTask task = spy( + new PipelineTask(instance, new PipelineInstanceNode(null, module), null)); Mockito.doReturn(taskId).when(task).getId(); - task.setTaskLogIndex(stepIndex); return task; } private File createAlgorithmTaskLog(long instanceId, long taskId, int jobIndex, int stepIndex) throws IOException { - PipelineTask task = createPipelineTask(instanceId, taskId, stepIndex); - String algorithmLogFilename = task.logFilename(jobIndex); + PipelineTask task = createPipelineTask(instanceId, taskId); + String algorithmLogFilename = pipelineTaskDataOperations.logFilename(task, jobIndex, + stepIndex); Files.createDirectories(DirectoryProperties.algorithmLogsDir()); File algorithmLogFile = DirectoryProperties.algorithmLogsDir() .resolve(algorithmLogFilename) diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/ZiggyMessengerTest.java b/src/test/java/gov/nasa/ziggy/services/messaging/ZiggyMessengerTest.java index f390116..581c0e5 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/ZiggyMessengerTest.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/ZiggyMessengerTest.java @@ -9,7 +9,7 @@ import java.util.Map; import java.util.concurrent.CountDownLatch; -import org.junit.After; +import org.junit.Before; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -26,11 +26,8 @@ */ public class ZiggyMessengerTest { - private final List stringsFromMessages = new ArrayList<>(); - - @After - public void tearDown() { - stringsFromMessages.clear(); + @Before + public void setUp() { ZiggyMessenger.reset(); } @@ -97,6 +94,8 @@ public void testPublishWithCountdownLatch() { @Test public void testTakeAction() { log.info("Start"); + List stringsFromMessages = new ArrayList<>(); + ZiggyMessenger.setStoreMessages(true); ZiggyMessenger.subscribe(Message1.class, message -> { diff --git a/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java b/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java index 28c702a..bc068b6 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java @@ -3,6 +3,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertThrows; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.doReturn; import java.util.ArrayList; import java.util.List; @@ -10,9 +11,11 @@ import org.junit.After; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.mockito.ArgumentMatchers; import org.mockito.Mockito; +import gov.nasa.ziggy.FlakyTestCategory; import gov.nasa.ziggy.module.ModuleFatalProcessingException; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; @@ -56,11 +59,11 @@ public void setup() { pipelineDefinitionOperations = Mockito.mock(PipelineDefinitionOperations.class); pipelineInstanceManager = Mockito.spy(PipelineInstanceManager.class); pipelineInstanceOperations = Mockito.mock(PipelineInstanceOperations.class); - Mockito.when(pipelineInstanceManager.pipelineInstanceOperations()) - .thenReturn(pipelineInstanceOperations); - Mockito.when(pipelineInstanceManager.pipelineExecutor()).thenReturn(pipelineExecutor); - Mockito.when(pipelineInstanceManager.pipelineDefinitionOperations()) - .thenReturn(pipelineDefinitionOperations); + doReturn(pipelineInstanceOperations).when(pipelineInstanceManager) + .pipelineInstanceOperations(); + doReturn(pipelineExecutor).when(pipelineInstanceManager).pipelineExecutor(); + doReturn(pipelineDefinitionOperations).when(pipelineInstanceManager) + .pipelineDefinitionOperations(); // Mock some other stuff pipelineDefinition = Mockito.mock(PipelineDefinition.class); @@ -331,9 +334,10 @@ public void testIntervals() { } /** - * Tests that the requested wait for the 2nd pipeline run, and the requested wait for - * completeion of a not-yet-finished pipeline run, work as expected + * Tests that the requested wait for the 2nd pipeline run, and the requested wait for completion + * of a not-yet-finished pipeline run, work as expected */ + @Category(FlakyTestCategory.class) @Test public void testWaiting() { StartPipelineRequest wftr = new StartPipelineRequest(PIPELINE_NAME, INSTANCE_NAME, @@ -365,6 +369,6 @@ public void testWaiting() { long dt = completeTimeMillis - startTimeMillis; // Make the interval long enough to tolerate some overhead time - assertTrue(dt > 85 && dt <= 105); + assertTrue("Complete time was" + dt + " milliseconds", dt > 85 && dt <= 105); } } diff --git a/src/test/java/gov/nasa/ziggy/supervisor/PipelineSupervisorTest.java b/src/test/java/gov/nasa/ziggy/supervisor/PipelineSupervisorTest.java index 282efa3..0eba2f7 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/PipelineSupervisorTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/PipelineSupervisorTest.java @@ -2,11 +2,12 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.mockito.Mockito.times; -import java.util.HashSet; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.Set; @@ -18,15 +19,16 @@ import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.QueueCommandManagerForUnitTests; import gov.nasa.ziggy.module.remote.QueueCommandManagerForUnitTests.QueueDeleteCommand; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; +import gov.nasa.ziggy.services.alert.Alert.Severity; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.services.messages.KilledTaskMessage; -import gov.nasa.ziggy.util.Requestor; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; +import gov.nasa.ziggy.services.messages.TaskHaltedMessage; /** * Unit tests for {@link PipelineSupervisor} class. @@ -38,8 +40,9 @@ * hope that someday we get the chance to backfill tests for legacy functionality. * * @author PT + * @author Bill Wohler */ -public class PipelineSupervisorTest implements Requestor { +public class PipelineSupervisorTest { @Rule public ZiggyPropertyRule queueCommandRule = new ZiggyPropertyRule( @@ -52,128 +55,133 @@ public class PipelineSupervisorTest implements Requestor { private PipelineSupervisor supervisor; private PipelineTaskOperations pipelineTaskOperations = Mockito .mock(PipelineTaskOperations.class); + private PipelineTaskDataOperations pipelineTaskDataOperations = Mockito + .mock(PipelineTaskDataOperations.class); private AlertService alertService = Mockito.mock(AlertService.class); - private Set stateFiles = new HashSet<>(); + private Map> jobIdsByTask = new HashMap<>(); + private List tasks = new ArrayList<>(); + + private PipelineTask pipelineTask1; + private PipelineTask pipelineTask2; + private PipelineTask pipelineTask3; + private PipelineTask pipelineTask4; @Before public void setUp() { supervisor = Mockito.spy(new PipelineSupervisor(1, 12000)); Mockito.doReturn(alertService).when(supervisor).alertService(); Mockito.doReturn(pipelineTaskOperations).when(supervisor).pipelineTaskOperations(); - Mockito.doReturn(stateFiles).when(supervisor).remoteTaskStateFiles(); + Mockito.doReturn(pipelineTaskDataOperations).when(supervisor).pipelineTaskDataOperations(); + Mockito.doReturn(jobIdsByTask).when(supervisor).jobIdsByTask(ArgumentMatchers.anyList()); + + pipelineTask1 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(1L).when(pipelineTask1).getId(); + pipelineTask2 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(2L).when(pipelineTask2).getId(); + pipelineTask3 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(3L).when(pipelineTask3).getId(); + pipelineTask4 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(4L).when(pipelineTask4).getId(); } - /** Verify that the task-kill command works when the state file collection is null. */ + /** Verify that the task-halt command works when no remote tasks are present. */ @Test - public void testKillTasksNullStateFileCollection() { - stateFiles = null; - supervisor.killRemoteTasks(KillTasksRequest.forTaskIds(this, Set.of(1L, 2L, 3L))); + public void testHaltTasksEmptyStateFileCollection() { + supervisor.haltRemoteTasks( + new HaltTasksRequest(Set.of(pipelineTask1, pipelineTask2, pipelineTask3))); assertEquals(0, queueCommandManager().getQueueDeleteCommands().size()); Mockito.verify(supervisor, times(0)) - .publishKilledTaskMessage(ArgumentMatchers.any(KillTasksRequest.class), - ArgumentMatchers.any(Long.class)); - } - - /** Verify that the task-kill command works when the state file collection is empty. */ - @Test - public void testKillTasksEmptyStateFileCollection() { - supervisor.killRemoteTasks(KillTasksRequest.forTaskIds(this, Set.of(1L, 2L, 3L))); - assertEquals(0, queueCommandManager().getQueueDeleteCommands().size()); - Mockito.verify(supervisor, times(0)) - .publishKilledTaskMessage(ArgumentMatchers.any(KillTasksRequest.class), - ArgumentMatchers.any(Long.class)); + .publishTaskHaltedMessage(ArgumentMatchers.any(PipelineTask.class)); } /** - * Verify that the kill-tasks command works when the state file collection and the kill request + * Verify that the halt-tasks command works when the state file collection and the halt request * have some overlap in tasks but not complete (specifically, each of them has a task that the * other one does not). */ @Test - public void killTasksAllSuccessful() { - StateFile stateFile1 = Mockito.mock(StateFile.class); - Mockito.when(stateFile1.getPipelineTaskId()).thenReturn(1L); - stateFiles.add(stateFile1); - StateFile stateFile2 = Mockito.mock(StateFile.class); - Mockito.when(stateFile2.getPipelineTaskId()).thenReturn(2L); - stateFiles.add(stateFile2); - StateFile stateFile3 = Mockito.mock(StateFile.class); - Mockito.when(stateFile3.getPipelineTaskId()).thenReturn(3L); - stateFiles.add(stateFile3); - - KillTasksRequest request = KillTasksRequest.forTaskIds(this, Set.of(1L, 2L, 4L)); - supervisor.killRemoteTasks(request); - Map queueDeleteCommands = queueCommandManager() + public void testHaltTasksAllSuccessful() { + + jobIdsByTask.put(pipelineTask1, List.of(100L, 101L)); + jobIdsByTask.put(pipelineTask2, List.of(102L, 103L)); + jobIdsByTask.put(pipelineTask3, List.of(105L, 106L)); + + tasks.add(pipelineTask1); + tasks.add(pipelineTask2); + tasks.add(pipelineTask4); + + HaltTasksRequest request = new HaltTasksRequest(tasks); + supervisor.haltRemoteTasks(request); + List queueDeleteCommands = queueCommandManager() .getQueueDeleteCommands(); assertEquals(2, queueDeleteCommands.size()); - assertNull(queueDeleteCommands.get(3L)); - assertNull(queueDeleteCommands.get(4L)); - assertEquals(stateFile1, queueDeleteCommands.get(1L).getStateFile()); - assertEquals(0, queueDeleteCommands.get(1L).getReturnCode()); - assertEquals(stateFile2, queueDeleteCommands.get(2L).getStateFile()); - assertEquals(0, queueDeleteCommands.get(2L).getReturnCode()); - Mockito.verify(supervisor).publishKilledTaskMessage(request, 1L); - Mockito.verify(supervisor).publishKilledTaskMessage(request, 2L); - Mockito.verify(supervisor, times(0)).publishKilledTaskMessage(request, 3L); - Mockito.verify(supervisor, times(0)).publishKilledTaskMessage(request, 4L); + QueueDeleteCommand queueDeleteCommand = queueDeleteCommands.get(0); + assertEquals(0, queueDeleteCommand.getReturnCode()); + assertTrue(queueDeleteCommand.getJobIds().contains(100L)); + assertTrue(queueDeleteCommand.getJobIds().contains(101L)); + assertEquals(2, queueDeleteCommand.getJobIds().size()); + queueDeleteCommand = queueDeleteCommands.get(1); + assertEquals(0, queueDeleteCommand.getReturnCode()); + assertTrue(queueDeleteCommand.getJobIds().contains(102L)); + assertTrue(queueDeleteCommand.getJobIds().contains(103L)); + assertEquals(2, queueDeleteCommand.getJobIds().size()); + Mockito.verify(supervisor).publishTaskHaltedMessage(pipelineTask1); + Mockito.verify(supervisor).publishTaskHaltedMessage(pipelineTask2); + Mockito.verify(supervisor, times(0)).publishTaskHaltedMessage(pipelineTask3); + Mockito.verify(supervisor, times(0)).publishTaskHaltedMessage(pipelineTask4); } /** - * Verify that the correct action is taken when not all the tasks to be killed are in fact - * successfully killed. + * Verify that the correct action is taken when not all the tasks to be halted are in fact + * successfully halted. */ @Test - public void testKillTasksSomeFailures() { - StateFile stateFile1 = Mockito.mock(StateFile.class); - Mockito.when(stateFile1.getPipelineTaskId()).thenReturn(1L); - stateFiles.add(stateFile1); - StateFile stateFile2 = Mockito.mock(StateFile.class); - Mockito.when(stateFile2.getPipelineTaskId()).thenReturn(150L); - stateFiles.add(stateFile2); - StateFile stateFile3 = Mockito.mock(StateFile.class); - Mockito.when(stateFile3.getPipelineTaskId()).thenReturn(3L); - stateFiles.add(stateFile3); + public void testHaltTasksSomeFailures() { + jobIdsByTask.put(pipelineTask1, List.of(100L, 101L)); + jobIdsByTask.put(pipelineTask2, List.of(102L, 203L)); + jobIdsByTask.put(pipelineTask3, List.of(105L, 106L)); + + tasks.add(pipelineTask1); + tasks.add(pipelineTask2); + tasks.add(pipelineTask4); // Two kill task requests should have gone out. - KillTasksRequest request = KillTasksRequest.forTaskIds(this, Set.of(1L, 150L, 4L)); - supervisor.killRemoteTasks(request); - Map queueDeleteCommands = queueCommandManager() + HaltTasksRequest request = new HaltTasksRequest(tasks); + supervisor.haltRemoteTasks(request); + List queueDeleteCommands = queueCommandManager() .getQueueDeleteCommands(); assertEquals(2, queueDeleteCommands.size()); - assertNull(queueDeleteCommands.get(3L)); - assertNull(queueDeleteCommands.get(4L)); - - // One of the two had the qdel command return 0, the other returned 1. - assertEquals(stateFile1, queueDeleteCommands.get(1L).getStateFile()); - assertEquals(0, queueDeleteCommands.get(1L).getReturnCode()); - assertEquals(stateFile2, queueDeleteCommands.get(150L).getStateFile()); - assertEquals(1, queueDeleteCommands.get(150L).getReturnCode()); + QueueDeleteCommand queueDeleteCommand = queueDeleteCommands.get(0); + assertEquals(0, queueDeleteCommand.getReturnCode()); + assertTrue(queueDeleteCommand.getJobIds().contains(100L)); + assertTrue(queueDeleteCommand.getJobIds().contains(101L)); + assertEquals(2, queueDeleteCommand.getJobIds().size()); + queueDeleteCommand = queueDeleteCommands.get(1); + assertEquals(1, queueDeleteCommand.getReturnCode()); + assertTrue(queueDeleteCommand.getJobIds().contains(102L)); + assertTrue(queueDeleteCommand.getJobIds().contains(203L)); + assertEquals(2, queueDeleteCommand.getJobIds().size()); // Only the case where qdel returned 0 should publish a task message. - Mockito.verify(supervisor).publishKilledTaskMessage(request, 1L); - Mockito.verify(supervisor, times(0)).publishKilledTaskMessage(request, 150L); - Mockito.verify(supervisor, times(0)).publishKilledTaskMessage(request, 3L); - Mockito.verify(supervisor, times(0)).publishKilledTaskMessage(request, 4L); + Mockito.verify(supervisor).publishTaskHaltedMessage(pipelineTask1); + Mockito.verify(supervisor, times(0)).publishTaskHaltedMessage(pipelineTask2); + Mockito.verify(supervisor, times(0)).publishTaskHaltedMessage(pipelineTask3); + Mockito.verify(supervisor, times(0)).publishTaskHaltedMessage(pipelineTask4); } - /** Verify that the correct action is taken when a {@link KilledTaskMessage} arrives. */ + /** Verify that the correct action is taken when a {@link TaskHaltedMessage} arrives. */ @Test - public void testHandleKilledTaskMessage() { - assertFalse(PipelineSupervisor.taskOnKilledTaskList(1L)); - KilledTaskMessage message = new KilledTaskMessage(this, 1L); - supervisor.handleKilledTaskMessage(message); - assertTrue(PipelineSupervisor.taskOnKilledTaskList(1L)); + public void testHandleHaltedTaskMessage() { + assertFalse(PipelineSupervisor.taskOnHaltedTaskList(pipelineTask1)); + TaskHaltedMessage message = new TaskHaltedMessage(pipelineTask1); + supervisor.handleTaskHaltedMessage(message); + assertTrue(PipelineSupervisor.taskOnHaltedTaskList(pipelineTask1)); Mockito.verify(alertService) - .generateAndBroadcastAlert("PI", 1L, AlertService.Severity.ERROR, "Task 1 halted"); - Mockito.verify(pipelineTaskOperations).taskErrored(1L); + .generateAndBroadcastAlert("PI", pipelineTask1, Severity.ERROR, "Task 1 halted"); + Mockito.verify(pipelineTaskDataOperations).taskErrored(pipelineTask1); } public QueueCommandManagerForUnitTests queueCommandManager() { - return (QueueCommandManagerForUnitTests) supervisor.getQueueCommandManager(); - } - - @Override - public Object requestorIdentifier() { - return Integer.valueOf(0); + return (QueueCommandManagerForUnitTests) supervisor.queueCommandManager(); } } diff --git a/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java b/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java index b7f344c..dbbe134 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java @@ -17,33 +17,42 @@ import org.junit.After; import org.junit.Before; import org.junit.Test; +import org.junit.experimental.categories.Category; import org.mockito.ArgumentMatchers; import org.mockito.Mockito; +import gov.nasa.ziggy.FlakyTestCategory; import gov.nasa.ziggy.TestEventDetector; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskCrud; +import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskDataOperations; import gov.nasa.ziggy.pipeline.definition.database.PipelineTaskOperations; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.database.DatabaseService; -import gov.nasa.ziggy.services.messages.KillTasksRequest; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; import gov.nasa.ziggy.services.messages.TaskRequest; -import gov.nasa.ziggy.util.Requestor; import gov.nasa.ziggy.worker.WorkerResources; /** * Unit tests for {@link TaskRequestHandlerLifecycleManager} class. * * @author PT + * @author Bill Wohler */ public class TaskRequestHandlerLifecycleManagerTest { private InstrumentedTaskRequestHandlerLifecycleManager lifecycleManager; private Thread taskRequestLoopThread; + private PipelineTask pipelineTask1; + private PipelineTask pipelineTask2; + private PipelineTask pipelineTask3; + private PipelineTask pipelineTask4; + @Before public void setUp() throws InterruptedException { DatabaseService.setInstance(Mockito.mock(DatabaseService.class)); @@ -56,6 +65,15 @@ public void setUp() throws InterruptedException { lifecycleManager.start(); }); taskRequestLoopThread.setDaemon(true); + + pipelineTask1 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(1L).when(pipelineTask1).getId(); + pipelineTask2 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(2L).when(pipelineTask2).getId(); + pipelineTask3 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(3L).when(pipelineTask3).getId(); + pipelineTask4 = Mockito.spy(PipelineTask.class); + Mockito.doReturn(4L).when(pipelineTask4).getId(); } @After @@ -93,7 +111,7 @@ public void testStart() { assertNull(lifecycleManager.getPipelineDefinitionNodeId()); // Add a task to the queue and wait for it to get handled. - lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, 1L, + lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, pipelineTask1, PipelineInstance.Priority.NORMAL, false, PipelineModule.RunMode.STANDARD)); TestEventDetector.detectTestEvent(500L, () -> allTaskRequestHandlers.size() > 0); // There should be one set of task request handlers constructed, with 1 handler in the set. @@ -134,7 +152,7 @@ public void testStart() { public void testShutdown() { taskRequestLoopThread.start(); - lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, 1L, + lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, pipelineTask1, PipelineInstance.Priority.NORMAL, false, PipelineModule.RunMode.STANDARD)); TestEventDetector.detectTestEvent(500L, () -> lifecycleManager.getTaskRequestHandlers().size() > 0); @@ -157,6 +175,7 @@ public void testShutdown() { *
  • The lifecycle manager's pipeline definition node ID is updated. *
      */ + @Category(FlakyTestCategory.class) @Test public void testDefinitionNodeTransition() { @@ -164,7 +183,7 @@ public void testDefinitionNodeTransition() { List> allTaskRequestHandlers = lifecycleManager .getTaskRequestHandlers(); - lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, 1L, + lifecycleManager.addTaskRequestToQueue(new TaskRequest(1L, 1L, -1L, pipelineTask1, PipelineInstance.Priority.NORMAL, false, PipelineModule.RunMode.STANDARD)); TestEventDetector.detectTestEvent(500L, () -> allTaskRequestHandlers.size() > 0); @@ -173,9 +192,12 @@ public void testDefinitionNodeTransition() { () -> lifecycleManager.getPipelineDefinitionNodeId() == -1L)); // Create some task requests - TaskRequest t1 = new TaskRequest(0, 0, -2, 1, Priority.NORMAL, false, RunMode.STANDARD); - TaskRequest t2 = new TaskRequest(0, 0, -2, 2, Priority.NORMAL, false, RunMode.STANDARD); - TaskRequest t3 = new TaskRequest(0, 0, -2, 3, Priority.NORMAL, false, RunMode.STANDARD); + TaskRequest t1 = new TaskRequest(0, 0, -2, pipelineTask1, Priority.NORMAL, false, + RunMode.STANDARD); + TaskRequest t2 = new TaskRequest(0, 0, -2, pipelineTask2, Priority.NORMAL, false, + RunMode.STANDARD); + TaskRequest t3 = new TaskRequest(0, 0, -2, pipelineTask3, Priority.NORMAL, false, + RunMode.STANDARD); lifecycleManager.setMaxWorkers(2); @@ -251,25 +273,54 @@ public void testRemoveRequestsFromQueue() { TestEventDetector.detectTestEvent(500L, () -> lifecycleManager.taskRequestSize() == 0)); // Put some tasks into the queue. - lifecycleManager.addTaskRequestToQueue( - new TaskRequest(0, 0, -2, 1, Priority.NORMAL, false, RunMode.STANDARD)); - lifecycleManager.addTaskRequestToQueue( - new TaskRequest(0, 0, -2, 2, Priority.NORMAL, false, RunMode.STANDARD)); - lifecycleManager.addTaskRequestToQueue( - new TaskRequest(0, 0, -2, 3, Priority.NORMAL, false, RunMode.STANDARD)); + TaskRequest taskRequest = new TaskRequest(0, 0, -2, pipelineTask1, Priority.NORMAL, false, + RunMode.STANDARD); + lifecycleManager.addTaskRequestToQueue(taskRequest); + taskRequest = new TaskRequest(0, 0, -2, pipelineTask2, Priority.NORMAL, false, + RunMode.STANDARD); + lifecycleManager.addTaskRequestToQueue(taskRequest); + taskRequest = new TaskRequest(0, 0, -2, pipelineTask3, Priority.NORMAL, false, + RunMode.STANDARD); + lifecycleManager.addTaskRequestToQueue(taskRequest); // Check that the tasks stayed put assertEquals(3, lifecycleManager.taskRequestSize()); - KillTasksRequest request = KillTasksRequest.forTaskIds(lifecycleManager, - List.of(1L, 3L, 4L)); - lifecycleManager.killQueuedTasksAction(request); + List pipelineTasks = List.of(pipelineTask1, pipelineTask3, pipelineTask4); + HaltTasksRequest request = new HaltTasksRequest(pipelineTasks); + lifecycleManager.haltQueuedTasksAction(request); + assertEquals(1, lifecycleManager.taskRequestSize()); + assertEquals(pipelineTask2, lifecycleManager.taskRequestQueuePeek().getPipelineTask()); + Mockito.verify(lifecycleManager).publishTaskHaltedMessage(pipelineTask1); + Mockito.verify(lifecycleManager, times(0)).publishTaskHaltedMessage(pipelineTask2); + Mockito.verify(lifecycleManager).publishTaskHaltedMessage(pipelineTask3); + Mockito.verify(lifecycleManager, times(0)).publishTaskHaltedMessage(pipelineTask4); + } + + @Test + public void testHandleQueuedTasksAction() { + + lifecycleManager = Mockito.spy(lifecycleManager); + // Shut off the task request handler threads so that the tasks don't instantly fly + // out of the queue. + assertTrue( + TestEventDetector.detectTestEvent(500L, () -> lifecycleManager.taskRequestSize() == 0)); + + // Put some tasks into the queue. + TaskRequest taskRequest1 = new TaskRequest(0, 0, -2, pipelineTask1, Priority.NORMAL, false, + RunMode.STANDARD); + TaskRequest taskRequest2 = new TaskRequest(0, 0, -2, pipelineTask2, Priority.NORMAL, false, + RunMode.STANDARD); + Mockito.when(lifecycleManager.pipelineTaskDataOperations().haltRequested(pipelineTask1)) + .thenReturn(false); + Mockito.when(lifecycleManager.pipelineTaskDataOperations().haltRequested(pipelineTask2)) + .thenReturn(true); + lifecycleManager.handleTaskRequestAction(taskRequest1); + lifecycleManager.handleTaskRequestAction(taskRequest2); assertEquals(1, lifecycleManager.taskRequestSize()); - assertEquals(2, lifecycleManager.taskRequestQueuePeek().getTaskId()); - Mockito.verify(lifecycleManager).publishKilledTaskMessage(request, 1L); - Mockito.verify(lifecycleManager, times(0)).publishKilledTaskMessage(request, 2L); - Mockito.verify(lifecycleManager).publishKilledTaskMessage(request, 3L); - Mockito.verify(lifecycleManager, times(0)).publishKilledTaskMessage(request, 4L); + assertEquals(pipelineTask1, lifecycleManager.taskRequestQueuePeek().getPipelineTask()); + Mockito.verify(lifecycleManager).publishTaskHaltedMessage(pipelineTask2); + Mockito.verify(lifecycleManager, times(0)).publishTaskHaltedMessage(pipelineTask1); } /** @@ -279,11 +330,12 @@ public void testRemoveRequestsFromQueue() { * @author PT */ public static class InstrumentedTaskRequestHandlerLifecycleManager - extends TaskRequestHandlerLifecycleManager implements Requestor { + extends TaskRequestHandlerLifecycleManager { private int maxWorkers; private PipelineTaskCrud pipelineTaskCrud; - private PipelineTaskOperations pipelineTaskOperations; + private PipelineTaskOperations mockedPipelineTaskOperations; + private PipelineTaskDataOperations mockedPipelineTaskDataOperations; public InstrumentedTaskRequestHandlerLifecycleManager() { super(true); @@ -291,7 +343,8 @@ public InstrumentedTaskRequestHandlerLifecycleManager() { pipelineTaskCrud = Mockito.mock(PipelineTaskCrud.class); Mockito.when(pipelineTaskCrud.retrieveAll(ArgumentMatchers. anyList())) .thenReturn(new ArrayList<>()); - pipelineTaskOperations = Mockito.spy(PipelineTaskOperations.class); + mockedPipelineTaskOperations = Mockito.spy(PipelineTaskOperations.class); + mockedPipelineTaskDataOperations = Mockito.mock(PipelineTaskDataOperations.class); } public void setMaxWorkers(int maxWorkers) { @@ -310,12 +363,12 @@ protected AlertService alertService() { @Override protected PipelineTaskOperations pipelineTaskOperations() { - return pipelineTaskOperations; + return mockedPipelineTaskOperations; } @Override - public Object requestorIdentifier() { - return Long.valueOf(0L); + public PipelineTaskDataOperations pipelineTaskDataOperations() { + return mockedPipelineTaskDataOperations; } } } diff --git a/src/test/java/gov/nasa/ziggy/supervisor/WorkerTaskPriorityTest.java b/src/test/java/gov/nasa/ziggy/supervisor/WorkerTaskPriorityTest.java index 54cd7e8..5bca15f 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/WorkerTaskPriorityTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/WorkerTaskPriorityTest.java @@ -1,31 +1,55 @@ package gov.nasa.ziggy.supervisor; import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; import java.util.concurrent.PriorityBlockingQueue; +import org.junit.Before; import org.junit.Test; import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.messages.TaskRequest; /** * Tests to ensure that task priority is correctly managed. * * @author PT + * @author Bill Wohler */ public class WorkerTaskPriorityTest { + private PipelineTask pipelineTask5; + private PipelineTask pipelineTask6; + private PipelineTask pipelineTask7; + private PipelineTask pipelineTask10; + + @Before + public void setUp() { + pipelineTask5 = spy(PipelineTask.class); + doReturn(5L).when(pipelineTask5).getId(); + pipelineTask6 = spy(PipelineTask.class); + doReturn(6L).when(pipelineTask6).getId(); + pipelineTask7 = spy(PipelineTask.class); + doReturn(7L).when(pipelineTask7).getId(); + pipelineTask10 = spy(PipelineTask.class); + doReturn(10L).when(pipelineTask10).getId(); + } + @Test public void testTaskPrioritization() throws InterruptedException { PriorityBlockingQueue queue = new PriorityBlockingQueue<>(); - TaskRequest w1 = new TaskRequest(100, 100, 70, 5, Priority.LOWEST, false, RunMode.STANDARD); - TaskRequest w2 = new TaskRequest(100, 100, 70, 7, Priority.HIGHEST, false, + TaskRequest w1 = new TaskRequest(100, 100, 70, pipelineTask5, Priority.LOWEST, false, + RunMode.STANDARD); + TaskRequest w2 = new TaskRequest(100, 100, 70, pipelineTask7, Priority.HIGHEST, false, + RunMode.STANDARD); + TaskRequest w3 = new TaskRequest(100, 100, 70, pipelineTask6, Priority.LOWEST, false, RunMode.STANDARD); - TaskRequest w3 = new TaskRequest(100, 100, 70, 6, Priority.LOWEST, false, RunMode.STANDARD); - TaskRequest w4 = new TaskRequest(100, 100, 69, 10, Priority.LOWEST, false, + TaskRequest w4 = new TaskRequest(100, 100, 69, pipelineTask10, Priority.LOWEST, false, RunMode.STANDARD); // Add the tasks in inverse-task-ID order diff --git a/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java b/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java index 8a46104..e12e38a 100644 --- a/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java +++ b/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java @@ -9,13 +9,13 @@ import org.junit.Test; import org.mockito.Mockito; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; public class InstanceCostEstimateDialogTest { @Test public void testInstanceCost() { - List pipelineTasks = List.of(createPipelineTask(0.000123)); + List pipelineTasks = List.of(createPipelineTask(0.000123)); assertEquals("0.0001", instanceCost(pipelineTasks)); pipelineTasks = List.of(createPipelineTask(1.00123)); assertEquals("1.001", instanceCost(pipelineTasks)); @@ -27,8 +27,8 @@ public void testInstanceCost() { assertEquals("1001.2", instanceCost(pipelineTasks)); } - private PipelineTask createPipelineTask(double costEstimate) { - PipelineTask pipelineTask = mock(PipelineTask.class); + private PipelineTaskDisplayData createPipelineTask(double costEstimate) { + PipelineTaskDisplayData pipelineTask = mock(PipelineTaskDisplayData.class); Mockito.when(pipelineTask.costEstimate()).thenReturn(costEstimate); return pipelineTask; } diff --git a/src/test/java/gov/nasa/ziggy/ui/status/WorkerStatusTableModelTest.java b/src/test/java/gov/nasa/ziggy/ui/status/WorkerStatusTableModelTest.java index 444c313..b5ca819 100644 --- a/src/test/java/gov/nasa/ziggy/ui/status/WorkerStatusTableModelTest.java +++ b/src/test/java/gov/nasa/ziggy/ui/status/WorkerStatusTableModelTest.java @@ -3,7 +3,9 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.spy; import static org.mockito.Mockito.times; +import static org.mockito.Mockito.when; import java.util.Iterator; @@ -11,6 +13,7 @@ import org.junit.Test; import org.mockito.Mockito; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.messages.WorkerStatusMessage; import gov.nasa.ziggy.ui.status.WorkerStatusPanel.WorkerStatusTableModel; @@ -18,17 +21,25 @@ * Unit tests for {@link WorkerStatusTableModel} class. * * @author PT + * @author Bill Wohler */ public class WorkerStatusTableModelTest { private WorkerStatusTableModel tableModel; + private PipelineTask pipelineTask2; + private PipelineTask pipelineTask3; @Before public void setUp() { // Set up a table model that doesn't try to redraw the (non existent) table. - tableModel = Mockito.spy(WorkerStatusTableModel.class); + tableModel = spy(WorkerStatusTableModel.class); Mockito.doNothing().when(tableModel).redrawTable(); + + pipelineTask2 = spy(PipelineTask.class); + when(pipelineTask2.getId()).thenReturn(2L); + pipelineTask3 = spy(PipelineTask.class); + when(pipelineTask3.getId()).thenReturn(3L); } @Test @@ -39,8 +50,8 @@ public void testAddMessage() { assertTrue(tableModel.messageSet().isEmpty()); // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); // There should be a message in the Map and in the Set. @@ -58,13 +69,13 @@ public void testAddMessage() { public void testReplaceMessage() { // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); // Add another message for the same task - message = new WorkerStatusMessage(1, "more awesome", "2", "3", "dummy", "single", 5678L, - false); + message = new WorkerStatusMessage(1, "more awesome", "2", pipelineTask3, "dummy", "single", + 5678L, false); tableModel.updateModel(message); // There should be a message in the Map and in the Set. @@ -89,13 +100,13 @@ public void testReplaceMessage() { public void testFinalMessage() { // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); // Send a final message for the same task. - message = new WorkerStatusMessage(1, "more awesome", "2", "3", "dummy", "single", 5678L, - true); + message = new WorkerStatusMessage(1, "more awesome", "2", pipelineTask3, "dummy", "single", + 5678L, true); tableModel.updateModel(message); // The model should now be empty. @@ -110,12 +121,13 @@ public void testFinalMessage() { public void testMessageOrdering() { // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); // Add a message for a task with a lower number. - message = new WorkerStatusMessage(1, "awesome", "2", "2", "dummy", "single", 1234L, false); + message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask2, "dummy", "single", + 1234L, false); tableModel.updateModel(message); // There should be 2 messages in the model. @@ -125,17 +137,17 @@ public void testMessageOrdering() { // The second message to be added should be first when iterating over the model. Iterator messageIterator = tableModel.messageSet().iterator(); message = messageIterator.next(); - assertEquals("2", message.getTaskId()); + assertEquals(pipelineTask2, message.getPipelineTask()); message = messageIterator.next(); - assertEquals("3", message.getTaskId()); + assertEquals(pipelineTask3, message.getPipelineTask()); } @Test public void testMessageOutdating() { // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); // On the first call, the message should be marked as outdated. @@ -156,8 +168,8 @@ public void testMessageOutdating() { public void testMessageLifeCycle() { // Add a not-final-message message. - WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", "3", "dummy", - "single", 1234L, false); + WorkerStatusMessage message = new WorkerStatusMessage(1, "awesome", "2", pipelineTask3, + "dummy", "single", 1234L, false); tableModel.updateModel(message); assertEquals(1, tableModel.statusMessages().size()); @@ -172,8 +184,8 @@ public void testMessageLifeCycle() { Mockito.verify(tableModel, times(1)).redrawTable(); // Send a new message from the same task. - message = new WorkerStatusMessage(1, "more awesome", "2", "3", "dummy", "single", 1234L, - false); + message = new WorkerStatusMessage(1, "more awesome", "2", pipelineTask3, "dummy", "single", + 1234L, false); tableModel.updateModel(message); assertEquals(1, tableModel.statusMessages().size()); assertTrue(tableModel.statusMessages().get(message)); @@ -188,8 +200,8 @@ public void testMessageLifeCycle() { Mockito.verify(tableModel, times(2)).redrawTable(); // Send a final message. - message = new WorkerStatusMessage(1, "more awesome", "2", "3", "dummy", "single", 1234L, - true); + message = new WorkerStatusMessage(1, "more awesome", "2", pipelineTask3, "dummy", "single", + 1234L, true); tableModel.updateModel(message); assertTrue(tableModel.statusMessages().isEmpty()); Mockito.verify(tableModel, times(3)).redrawTable(); diff --git a/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkTest.java b/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkTest.java new file mode 100644 index 0000000..b261841 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkTest.java @@ -0,0 +1,130 @@ +package gov.nasa.ziggy.uow; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.fail; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.TreeSet; + +import org.junit.Test; + +import gov.nasa.ziggy.pipeline.definition.Parameter; + +public class UnitOfWorkTest { + + private static final Collection PARAMETERS = Set.of( + new Parameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, "brief1"), + new Parameter("C", "valueC"), new Parameter("B", "valueB"), new Parameter("A", "valueA")); + + @Test + public void testBriefState() { + assertEquals("brief1", createUnitOfWork().briefState()); + } + + @Test + public void testSetBriefState() { + try { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.setBriefState(null); + fail("Expected NullPointerException"); + } catch (NullPointerException expected) { + } + try { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.setBriefState(" "); + fail("Expected IllegalArgumentException"); + } catch (IllegalArgumentException expected) { + } + + UnitOfWork unitOfWork = createUnitOfWork(); + unitOfWork.setBriefState("brief2"); + assertEquals("brief2", unitOfWork.briefState()); + } + + @Test + public void testAddParameter() { + try { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.addParameter(null); + fail("Expected NullPointerException"); + } catch (NullPointerException expected) { + } + + UnitOfWork unitOfWork = createUnitOfWork(); + Parameter parameter = new Parameter("new parameter", "parameter value"); + unitOfWork.addParameter(parameter); + verifyParameter(parameter, unitOfWork.getParameter(parameter.getName())); + } + + @Test + public void testGetParameter() { + UnitOfWork unitOfWork = createUnitOfWork(); + assertEquals(null, unitOfWork.getParameter(null)); + for (Parameter parameter : PARAMETERS) { + verifyParameter(parameter, unitOfWork.getParameter(parameter.getName())); + } + } + + @Test + public void testGetParameters() { + UnitOfWork unitOfWork = new UnitOfWork(); + assertEquals(0, unitOfWork.getParameters().size()); + + unitOfWork = createUnitOfWork(); + assertEquals(4, unitOfWork.getParameters().size()); + assertEquals(PARAMETERS, unitOfWork.getParameters()); + } + + @Test + public void testSetParameters() { + try { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.setParameters(null); + fail("Expected NullPointerException"); + } catch (NullPointerException expected) { + } + try { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.setParameters(new HashSet<>()); + fail("Expected IllegalArgumentException"); + } catch (IllegalArgumentException expected) { + } + + Collection newParameters = Set.of( + new Parameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, "brief2"), + new Parameter("CC", "valueCC"), new Parameter("BB", "valueBB"), + new Parameter("AA", "valueAA")); + UnitOfWork unitOfWork = createUnitOfWork(); + unitOfWork.setParameters(newParameters); + assertNotEquals(PARAMETERS, unitOfWork.getParameters()); + assertEquals(newParameters, unitOfWork.getParameters()); + } + + @Test + public void testCompareTo() { + UnitOfWork unitOfWork = createUnitOfWork(); + List parameters = new ArrayList<>(new TreeSet<>(unitOfWork.getParameters())); + assertEquals(4, parameters.size()); + assertEquals("valueA", parameters.get(0).getValue()); + assertEquals("valueB", parameters.get(1).getValue()); + assertEquals("valueC", parameters.get(2).getValue()); + assertEquals("brief1", parameters.get(3).getValue()); + } + + private UnitOfWork createUnitOfWork() { + UnitOfWork unitOfWork = new UnitOfWork(); + unitOfWork.setParameters(PARAMETERS); + return unitOfWork; + } + + private void verifyParameter(Parameter expectedParameter, Parameter actualParameter) { + assertEquals(expectedParameter, actualParameter); + assertEquals(expectedParameter.getName(), actualParameter.getName()); + assertEquals(expectedParameter.getValue(), actualParameter.getValue()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/BuildInfoTest.java b/src/test/java/gov/nasa/ziggy/util/BuildInfoTest.java new file mode 100644 index 0000000..3c2a3cd --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/BuildInfoTest.java @@ -0,0 +1,151 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.NoSuchElementException; + +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.util.BuildInfo.BuildType; + +/** Unit tests for {@link BuildInfo} class. */ +public class BuildInfoTest { + + public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); + public ZiggyPropertyRule ziggyHomePropertyRule = new ZiggyPropertyRule( + PropertyName.ZIGGY_HOME_DIR.property(), directoryRule, "ziggy"); + public ZiggyPropertyRule pipelineHomePropertyRule = new ZiggyPropertyRule( + PropertyName.PIPELINE_HOME_DIR.property(), directoryRule, "pipeline"); + + @Rule + public RuleChain ruleChain = RuleChain.outerRule(directoryRule) + .around(ziggyHomePropertyRule) + .around(pipelineHomePropertyRule); + + @Test + public void testWriteZiggyBuildFile() throws IOException { + writeSampleZiggyVersionFile(true); + Path buildFilePath = directoryRule.directory() + .resolve("ziggy") + .resolve("etc") + .resolve("ziggy-build.properties"); + assertTrue(Files.isRegularFile(buildFilePath)); + List properties = Files.readAllLines(buildFilePath); + assertTrue(properties.contains("ziggy.version = Alper")); + assertTrue(properties.contains("ziggy.version.branch = Bethe")); + assertTrue(properties.contains("ziggy.version.commit = Gamow")); + } + + @Test + public void testGetZiggySoftwareVersion() { + writeSampleZiggyVersionFile(true); + assertEquals("Alper", new BuildInfo(BuildType.ZIGGY).getSoftwareVersion()); + } + + @Test(expected = NoSuchElementException.class) + public void testErrorForMissingZiggyVersion() { + new BuildInfo(BuildType.ZIGGY).getSoftwareVersion(); + } + + @Test + public void testGetZiggyBranch() { + writeSampleZiggyVersionFile(true); + assertEquals("Bethe", new BuildInfo(BuildType.ZIGGY).getBranch()); + } + + @Test(expected = NoSuchElementException.class) + public void testErrorForMissingZiggyBranch() { + new BuildInfo(BuildType.ZIGGY).getBranch(); + } + + @Test + public void testGetZiggyCommit() { + writeSampleZiggyVersionFile(true); + assertEquals("Gamow", new BuildInfo(BuildType.ZIGGY).getRevision()); + } + + @Test(expected = NoSuchElementException.class) + public void testErrorForMissingZiggyCommit() { + new BuildInfo(BuildType.ZIGGY).getRevision(); + } + + @Test + public void testRelease() { + writeSampleZiggyVersionFile(true); + assertTrue(new BuildInfo(BuildType.ZIGGY).isRelease()); + } + + @Test + public void testNotRelease() { + writeSampleZiggyVersionFile(false); + assertFalse(new BuildInfo(BuildType.ZIGGY).isRelease()); + } + + @Test + public void testWritePipelineBuildFile() throws IOException { + writeSamplePipelineVersionFile(); + Path buildFilePath = directoryRule.directory() + .resolve("pipeline") + .resolve("etc") + .resolve("pipeline-build.properties"); + assertTrue(Files.exists(buildFilePath)); + assertTrue(Files.isRegularFile(buildFilePath)); + List properties = Files.readAllLines(buildFilePath); + assertTrue(properties.contains("pipeline.version = Alper")); + assertTrue(properties.contains("pipeline.version.branch = Bethe")); + assertTrue(properties.contains("pipeline.version.commit = Gamow")); + } + + @Test + public void testPipelineVersion() { + BuildInfo pipelineInfo = new BuildInfo(BuildType.PIPELINE); + assertEquals(BuildInfo.MISSING_PROPERTY_VALUE, pipelineInfo.getSoftwareVersion()); + assertEquals(BuildInfo.MISSING_PROPERTY_VALUE, pipelineInfo.getBranch()); + assertEquals(BuildInfo.MISSING_PROPERTY_VALUE, pipelineInfo.getRevision()); + writeSamplePipelineVersionFile(); + assertEquals("Alper", pipelineInfo.getSoftwareVersion()); + assertEquals("Bethe", pipelineInfo.getBranch()); + assertEquals("Gamow", pipelineInfo.getRevision()); + } + + private void writeSampleZiggyVersionFile(boolean release) { + String versionString = release ? "Alper" : "Alper-Gamow"; + BuildInfo ziggyVersion = Mockito.spy(new BuildInfo(BuildType.ZIGGY)); + Mockito.doReturn(List.of(versionString)) + .when(ziggyVersion) + .externalProcessResults(BuildInfo.VERSION_ARGS); + Mockito.doReturn(List.of("Bethe")) + .when(ziggyVersion) + .externalProcessResults(BuildInfo.BRANCH_ARGS); + Mockito.doReturn(List.of("Gamow")) + .when(ziggyVersion) + .externalProcessResults(BuildInfo.COMMIT_ARGS); + ziggyVersion.writeBuildFile(); + } + + private void writeSamplePipelineVersionFile() { + BuildInfo pipelineVersion = Mockito.spy(new BuildInfo(BuildType.PIPELINE)); + Mockito.doReturn(List.of("Alper")) + .when(pipelineVersion) + .externalProcessResults(BuildInfo.VERSION_ARGS); + Mockito.doReturn(List.of("Bethe")) + .when(pipelineVersion) + .externalProcessResults(BuildInfo.BRANCH_ARGS); + Mockito.doReturn(List.of("Gamow")) + .when(pipelineVersion) + .externalProcessResults(BuildInfo.COMMIT_ARGS); + pipelineVersion.writeBuildFile(); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java b/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java index 65b0fd8..9158a81 100644 --- a/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java +++ b/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java @@ -22,8 +22,9 @@ public void testFilterToList() { @Test public void testFilterToSet() { - Set set = Set.of(new Object(), new Date(), Double.valueOf(1.0), Integer.valueOf(1)); - assertEquals(Set.of(new Date()), CollectionFilters.filterToSet(set, Date.class)); + Set set = Set.of(new Object(), new Date(1000L), Double.valueOf(1.0), + Integer.valueOf(1)); + assertEquals(Set.of(new Date(1000L)), CollectionFilters.filterToSet(set, Date.class)); assertEquals(Set.of(Double.valueOf(1.0), Integer.valueOf(1)), CollectionFilters.filterToSet(set, Number.class)); } diff --git a/src/test/java/gov/nasa/ziggy/util/ZiggyUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/ZiggyUtilsTest.java new file mode 100644 index 0000000..2d8dcdc --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/ZiggyUtilsTest.java @@ -0,0 +1,59 @@ +package gov.nasa.ziggy.util; + +import static gov.nasa.ziggy.util.ZiggyUtils.tryPatiently; +import static org.junit.Assert.assertEquals; + +import org.junit.Before; +import org.junit.Test; + +import gov.nasa.ziggy.module.PipelineException; + +/** + * Tests the {@link ZiggyUtils} class. + * + * @author Bill Wohler + */ +public class ZiggyUtilsTest { + + int value; + + @Before + public void setUp() { + value = 0; + } + + @Test + public void testTryPatiently() { + assertEquals(null, tryPatiently("foo", 0, 1, () -> 42)); + assertEquals(42, tryPatiently("foo", 1, 1, () -> 42).intValue()); + + int loopCount = 4; + assertEquals(loopCount, tryPatiently("foo", loopCount, 1, () -> { + if (value++ < loopCount - 1) { + throw new Exception("test"); + } + return value; + }).intValue()); + } + + @Test(expected = NullPointerException.class) + public void testTryPatientlyWithNullMessage() { + tryPatiently(null, 1, 1, null); + } + + @Test(expected = NullPointerException.class) + public void testTryPatientlyWithNullSupplier() { + tryPatiently("foo", 1, 1, null); + } + + @Test(expected = PipelineException.class) + public void testTryPatientlyWithException() { + int loopCount = 4; + assertEquals(loopCount, tryPatiently("foo", loopCount - 1, 1, () -> { + if (value++ < loopCount - 1) { + throw new Exception("test"); + } + return value; + }).intValue()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java b/src/test/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModelTest.java similarity index 54% rename from src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java rename to src/test/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModelTest.java index 3409853..e3c1985 100644 --- a/src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java +++ b/src/test/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModelTest.java @@ -1,15 +1,22 @@ -package gov.nasa.ziggy.util; +package gov.nasa.ziggy.util.dispmod; import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.doReturn; import java.util.Date; import java.util.List; import org.junit.Test; +import org.mockito.Mockito; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskData; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; +import gov.nasa.ziggy.uow.UnitOfWork; +import gov.nasa.ziggy.util.SystemProxy; +import gov.nasa.ziggy.util.dispmod.PipelineStatsDisplayModel.TaskProcessingTimeStats; -public class TaskProcessingTimeStatsTest { +public class PipelineStatsDisplayModelTest { private static final long START_MILLIS = 1700000000000L; private static final long HOUR_MILLIS = 60 * 60 * 1000; @@ -20,17 +27,15 @@ public void test() { .of(pipelineTasks()); assertEquals(3, taskProcessingTimeStats.getCount()); assertEquals(2.0, taskProcessingTimeStats.getMax(), 0.0001); - assertEquals(new Date(START_MILLIS + 5 * HOUR_MILLIS), taskProcessingTimeStats.getMaxEnd()); assertEquals(1.0, taskProcessingTimeStats.getMean(), 0.0001); assertEquals(0.0, taskProcessingTimeStats.getMin(), 0.0001); assertEquals(new Date(START_MILLIS), taskProcessingTimeStats.getMinStart()); // TODO Calculator.net says stddev of 0, 1, 2 is 0.81649658092773, not 1.0 assertEquals(1.0, taskProcessingTimeStats.getStddev(), 0.0001); assertEquals(3.0, taskProcessingTimeStats.getSum(), 0.0001); - assertEquals(5.0, taskProcessingTimeStats.getTotalElapsed(), 0.0001); } - private List pipelineTasks() { + private List pipelineTasks() { // The first task took two hours and the second task started after the first and took one // hour. return List.of( @@ -41,10 +46,19 @@ private List pipelineTasks() { pipelineTask("module3", new Date(0), new Date(0))); } - private PipelineTask pipelineTask(String moduleName, Date start, Date end) { - PipelineTask pipelineTask = new PipelineTask(); - pipelineTask.setStartProcessingTime(start); - pipelineTask.setEndProcessingTime(end); - return pipelineTask; + private PipelineTaskDisplayData pipelineTask(String moduleName, Date start, Date end) { + PipelineTask pipelineTask = Mockito.mock(PipelineTask.class); + doReturn((long) 1).when(pipelineTask).getId(); + doReturn(start).when(pipelineTask).getCreated(); + doReturn(new UnitOfWork(moduleName)).when(pipelineTask).getUnitOfWork(); + + PipelineTaskDisplayData pipelineTaskDisplayData = new PipelineTaskDisplayData( + new PipelineTaskData(pipelineTask)); + SystemProxy.setUserTime(start.getTime()); + pipelineTaskDisplayData.getExecutionClock().start(); + SystemProxy.setUserTime(end.getTime()); + pipelineTaskDisplayData.getExecutionClock().stop(); + + return pipelineTaskDisplayData; } } diff --git a/src/test/java/gov/nasa/ziggy/util/dispmod/TaskSummaryDisplayModelTest.java b/src/test/java/gov/nasa/ziggy/util/dispmod/TaskSummaryDisplayModelTest.java index b1d48fd..9a20049 100644 --- a/src/test/java/gov/nasa/ziggy/util/dispmod/TaskSummaryDisplayModelTest.java +++ b/src/test/java/gov/nasa/ziggy/util/dispmod/TaskSummaryDisplayModelTest.java @@ -6,7 +6,7 @@ import org.junit.Test; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskDisplayData; import gov.nasa.ziggy.pipeline.definition.TaskCounts; import gov.nasa.ziggy.pipeline.definition.database.PipelineOperationsTestUtils; @@ -15,8 +15,9 @@ public class TaskSummaryDisplayModelTest { @Test public void test() { PipelineOperationsTestUtils pipelineOperationsTestUtils = new PipelineOperationsTestUtils(); - pipelineOperationsTestUtils.setUpFivePipelineTasks(); - List pipelineTasks = pipelineOperationsTestUtils.getPipelineTasks(); + pipelineOperationsTestUtils.setUpFivePipelineTaskDisplayData(); + List pipelineTasks = pipelineOperationsTestUtils + .getPipelineTaskDisplayData(); TaskCounts taskCounts = new TaskCounts(pipelineTasks); TaskSummaryDisplayModel model = new TaskSummaryDisplayModel(taskCounts); assertEquals(6, model.getColumnCount()); diff --git a/src/test/java/gov/nasa/ziggy/util/os/CpuInfoTest.java b/src/test/java/gov/nasa/ziggy/util/os/CpuInfoTest.java index 0363b66..b44842b 100644 --- a/src/test/java/gov/nasa/ziggy/util/os/CpuInfoTest.java +++ b/src/test/java/gov/nasa/ziggy/util/os/CpuInfoTest.java @@ -16,7 +16,7 @@ public class CpuInfoTest { @Test public void test() throws Exception { - CpuInfo cpuInfo = OperatingSystemType.getInstance().getCpuInfo(); + CpuInfo cpuInfo = OperatingSystemType.newInstance().getCpuInfo(); int numCores = cpuInfo.getNumCores(); diff --git a/src/test/java/gov/nasa/ziggy/util/os/MemInfoTest.java b/src/test/java/gov/nasa/ziggy/util/os/MemInfoTest.java index b0a92d0..998600a 100644 --- a/src/test/java/gov/nasa/ziggy/util/os/MemInfoTest.java +++ b/src/test/java/gov/nasa/ziggy/util/os/MemInfoTest.java @@ -12,7 +12,7 @@ public class MemInfoTest { @Test public void test() throws Exception { - MemInfo memInfo = OperatingSystemType.getInstance().getMemInfo(); + MemInfo memInfo = OperatingSystemType.newInstance().getMemInfo(); long totalMemory = memInfo.getTotalMemoryKB(); long freeMemory = memInfo.getFreeMemoryKB(); diff --git a/src/test/java/gov/nasa/ziggy/util/os/OperatingSystemTypeTest.java b/src/test/java/gov/nasa/ziggy/util/os/OperatingSystemTypeTest.java index 7f91973..f0fad7a 100644 --- a/src/test/java/gov/nasa/ziggy/util/os/OperatingSystemTypeTest.java +++ b/src/test/java/gov/nasa/ziggy/util/os/OperatingSystemTypeTest.java @@ -9,7 +9,7 @@ * @author Miles Cote */ public class OperatingSystemTypeTest { - private final OperatingSystemType operatingSystemType = OperatingSystemType.getInstance(); + private final OperatingSystemType operatingSystemType = OperatingSystemType.newInstance(); @Test public void testGetName() { @@ -63,13 +63,13 @@ public void testByNameWithEmptyName() { assertEquals(OperatingSystemType.DEFAULT, OperatingSystemType.byName("")); } - @Test(expected = IllegalArgumentException.class) + @Test(expected = NullPointerException.class) public void testByNameWithNullName() { OperatingSystemType.byName(null); } @Test public void testGetInstance() { - assertNotNull(OperatingSystemType.getInstance()); + assertNotNull(OperatingSystemType.newInstance()); } } diff --git a/src/test/java/gov/nasa/ziggy/util/os/ProcInfoTest.java b/src/test/java/gov/nasa/ziggy/util/os/ProcInfoTest.java index b1e202c..2b95f0e 100644 --- a/src/test/java/gov/nasa/ziggy/util/os/ProcInfoTest.java +++ b/src/test/java/gov/nasa/ziggy/util/os/ProcInfoTest.java @@ -1,5 +1,7 @@ package gov.nasa.ziggy.util.os; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; @@ -7,8 +9,6 @@ import org.junit.Test; import org.junit.experimental.categories.Category; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import gov.nasa.ziggy.IntegrationTestCategory; @@ -17,37 +17,31 @@ * * @author Forrest Girouard */ -@Category(IntegrationTestCategory.class) public class ProcInfoTest { - private static final Logger log = LoggerFactory.getLogger(ProcInfoTest.class); - + @Category(IntegrationTestCategory.class) @Test public void testChildPids() throws Exception { - List childPids = OperatingSystemType.getInstance().getProcInfo(1).getChildPids(); + List childPids = OperatingSystemType.newInstance().getProcInfo(1).getChildPids(); assertNotNull("childPids null", childPids); - - log.info("found " + childPids.size() + " children"); - assertTrue("At least one child", childPids.size() > 0); } + @Category(IntegrationTestCategory.class) @Test public void testChildPidsNoSuchName() throws Exception { - List childPids = OperatingSystemType.getInstance() + List childPids = OperatingSystemType.newInstance() .getProcInfo(1) .getChildPids("NoSuchName"); assertNotNull("childPids null", childPids); - - log.info("found " + childPids.size() + " children"); - assertTrue("No children", childPids.size() == 0); } + @Category(IntegrationTestCategory.class) @Test public void testChildPidsByName() throws Exception { - OperatingSystemType osType = OperatingSystemType.getInstance(); + OperatingSystemType osType = OperatingSystemType.newInstance(); String osName = osType.getName(); String processName; if (osName.equals(OperatingSystemType.MAC_OS_X.getName())) { @@ -58,47 +52,30 @@ public void testChildPidsByName() throws Exception { List childPids = osType.getProcInfo(1).getChildPids(processName); assertNotNull("childPids null", childPids); - - log.info("found " + childPids.size() + " children"); - assertTrue("At least one child", childPids.size() > 0); } @Test public void testParentPid() throws Exception { - long parentPid = OperatingSystemType.getInstance().getProcInfo(1).getParentPid(); - - log.info(String.format("found parent pid of %d for pid %d", parentPid, 1)); - - assertTrue("unexpected parent pid", parentPid == 0); + assertEquals(0, OperatingSystemType.newInstance().getProcInfo(1).getParentPid()); } @Test public void testPPid() throws Exception { - ProcInfo procInfo = OperatingSystemType.getInstance() + ProcInfo procInfo = OperatingSystemType.newInstance() .getProcInfo(gov.nasa.ziggy.util.os.ProcessUtils.getPid()); - - log.info(String.format("found parent pid of %d for pid %d", procInfo.getParentPid(), - procInfo.getPid())); - - assertTrue("unexpected parent pid", procInfo.getPid() != procInfo.getParentPid()); + assertNotEquals("Unexpected parent pid", procInfo.getPid(), procInfo.getParentPid()); } @Test public void testOpenFileLimit() throws Exception { - int limit = OperatingSystemType.getInstance().getProcInfo().getOpenFileLimit(); - - log.info("max open file limit is " + limit); - - assertTrue("max open files", limit > 0); + int limit = OperatingSystemType.newInstance().getProcInfo().getOpenFileLimit(); + assertTrue("Max open files", limit > 0); } @Test public void testMaximumPid() throws Exception { - long maxPid = OperatingSystemType.getInstance().getProcInfo().getMaximumPid(); - - log.info("maximum pid value is " + maxPid); - - assertTrue("maximum pid vale", maxPid > 0); + long maxPid = OperatingSystemType.newInstance().getProcInfo().getMaximumPid(); + assertTrue("Maximum pid value", maxPid > 0); } } diff --git a/src/test/java/gov/nasa/ziggy/util/os/ProcessUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/os/ProcessUtilsTest.java index b1be05e..05aeec6 100644 --- a/src/test/java/gov/nasa/ziggy/util/os/ProcessUtilsTest.java +++ b/src/test/java/gov/nasa/ziggy/util/os/ProcessUtilsTest.java @@ -39,7 +39,7 @@ public void cleanUpProcesses() { public void testPid() throws Exception { long pid = ProcessUtils.getPid(); assertTrue(pid > 0); // 0 is init on UNIX - assertTrue(pid <= OperatingSystemType.getInstance().getProcInfo().getMaximumPid()); + assertTrue(pid <= OperatingSystemType.newInstance().getProcInfo().getMaximumPid()); ExternalProcess psProcess = ExternalProcess .simpleExternalProcess("/bin/ps -o pid,comm " + Long.toString(pid)); psProcess.execute(); diff --git a/src/test/java/gov/nasa/ziggy/worker/PipelineWorkerTest.java b/src/test/java/gov/nasa/ziggy/worker/PipelineWorkerTest.java index eddb0b7..93e1e9b 100644 --- a/src/test/java/gov/nasa/ziggy/worker/PipelineWorkerTest.java +++ b/src/test/java/gov/nasa/ziggy/worker/PipelineWorkerTest.java @@ -2,6 +2,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNull; +import static org.mockito.Mockito.spy; import static org.mockito.Mockito.times; import java.util.List; @@ -10,8 +11,8 @@ import org.junit.Test; import org.mockito.Mockito; -import gov.nasa.ziggy.services.messages.KillTasksRequest; -import gov.nasa.ziggy.util.Requestor; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.messages.HaltTasksRequest; import gov.nasa.ziggy.util.SystemProxy; /** @@ -22,44 +23,58 @@ * functionality. * * @author PT + * @author Bill Wohler */ -public class PipelineWorkerTest implements Requestor { +public class PipelineWorkerTest { - private PipelineWorker worker = Mockito.spy(new PipelineWorker("dummy", 0)); - private KillTasksRequest request = KillTasksRequest.forTaskIds(this, List.of(1L, 2L, 3L)); + private PipelineTask pipelineTask1; + private PipelineTask pipelineTask2; + private PipelineTask pipelineTask3; + private PipelineTask pipelineTask100; + + private HaltTasksRequest request; @Before public void setUp() { SystemProxy.disableExit(); - Mockito.doNothing() - .when(worker) - .sendKilledTaskMessage(Mockito.any(KillTasksRequest.class), Mockito.anyLong()); + + pipelineTask1 = spy(PipelineTask.class); + pipelineTask2 = spy(PipelineTask.class); + pipelineTask3 = spy(PipelineTask.class); + pipelineTask100 = spy(PipelineTask.class); + + List pipelineTasks = List.of(pipelineTask1, pipelineTask2, pipelineTask3); + request = new HaltTasksRequest(pipelineTasks); } /** - * Exercises the {@link PipelineWorker#killTasks(KillTasksRequest)} method when the kill-tasks + * Exercises the {@link PipelineWorker#haltTask(HaltTasksRequest)} method when the kill-tasks * request doesn't include the task of the worker. */ @Test - public void testKillRequestOtherTasks() { - worker.setTaskId(100L); - worker.killTasks(request); - Mockito.verify(worker, times(0)).sendKilledTaskMessage(request, 100L); + public void testHaltRequestOtherTasks() { + PipelineWorker worker = createPipelineWorker(pipelineTask100); + worker.haltTask(request); + Mockito.verify(worker, times(0)).sendTaskHaltedMessage(request, pipelineTask100); Mockito.verify(worker, times(0)).killWorker(); assertNull(SystemProxy.getLatestExitCode()); } @Test - public void testKillRequestThisTask() { - worker.setTaskId(3L); - worker.killTasks(request); - Mockito.verify(worker).sendKilledTaskMessage(request, 3L); + public void testHaltRequestThisTask() { + PipelineWorker worker = createPipelineWorker(pipelineTask3); + worker.haltTask(request); + Mockito.verify(worker).sendTaskHaltedMessage(request, pipelineTask3); Mockito.verify(worker).killWorker(); assertEquals(0, SystemProxy.getLatestExitCode().intValue()); } - @Override - public Object requestorIdentifier() { - return Integer.valueOf(100); + private PipelineWorker createPipelineWorker(PipelineTask pipelineTask) { + PipelineWorker worker = spy(new PipelineWorker("dummy", pipelineTask, 0)); + Mockito.doNothing() + .when(worker) + .sendTaskHaltedMessage(Mockito.any(HaltTasksRequest.class), + Mockito.any(PipelineTask.class)); + return worker; } } diff --git a/src/test/matlab/ZiggyErrorWriterTest.m b/src/test/matlab/ZiggyErrorWriterTest.m index 34bd3e7..cf194e0 100644 --- a/src/test/matlab/ZiggyErrorWriterTest.m +++ b/src/test/matlab/ZiggyErrorWriterTest.m @@ -8,7 +8,7 @@ properties (GetAccess = 'public', SetAccess = 'private') origDir = []; taskDir = []; - subTaskDir = []; + subtaskDir = []; tempDir = []; end % properties @@ -21,10 +21,10 @@ function get_temp_directories(obj) obj.tempDir = tempdir(); obj.taskDir = fullfile(obj.tempDir, 'cal-50-100'); mkdir(obj.taskDir); - obj.subTaskDir = fullfile(obj.taskDir, 'st-0'); - mkdir(obj.subTaskDir); + obj.subtaskDir = fullfile(obj.taskDir, 'st-0'); + mkdir(obj.subtaskDir); obj.origDir = pwd; - cd(obj.subTaskDir); + cd(obj.subtaskDir); end end % TestMethodSetup methods @@ -36,7 +36,7 @@ function get_temp_directories(obj) % cd back to the original dir and delete the temp dirs function delete_temp_directories(obj) cd(obj.origDir); - rmdir(obj.subTaskDir); + rmdir(obj.subtaskDir); rmdir(obj.taskDir); end @@ -61,7 +61,7 @@ function f3(obj) function perform_error_file_checks(obj, expectedFileName, highestCaller, ... highestCallerLineNumber) - expectedFile = fullfile(obj.subTaskDir, expectedFileName); + expectedFile = fullfile(obj.subtaskDir, expectedFileName); obj.verifyEqual( exist(expectedFile, 'file'), 2, ... 'Error file does not exist in sub task directory!' ); h = hdf5ConverterClass(); diff --git a/test/data/PbsLogParser/pbs-log-comment-and-status b/test/data/PbsLogParser/pbs-log-comment-and-status new file mode 100644 index 0000000..a683e61 --- /dev/null +++ b/test/data/PbsLogParser/pbs-log-comment-and-status @@ -0,0 +1,40 @@ +Job 20255559.pbspl1.nas.nasa.gov started on Wed Aug 07 12:00:12 PDT 2024 +The job requested the following resources: + ncpus=1 + place=scatter:excl + walltime=00:30:00 + +PBS set the following environment variables: + FORT_BUFFERED = 1 + TZ = PST8PDT + +On r137i4n6: +SLF4J(I): Connected with provider of type [org.apache.logging.slf4j.SLF4JServiceProvider] +=>> PBS: job killed: walltime 1818 exceeded limit 1800 +Exception in thread "Thread-1" java.lang.IllegalMonitorStateException + at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372) + at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1007) + at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147) + at gov.nasa.ziggy.util.io.LockManager.releaseWriteLockInternal(LockManager.java:243) + at gov.nasa.ziggy.util.io.LockManager.releaseWriteLock(LockManager.java:135) + at gov.nasa.ziggy.util.io.LockManager.releaseAllLocks(LockManager.java:44) + at gov.nasa.ziggy.util.io.LockManager.lambda$new$0(LockManager.java:31) + at gov.nasa.ziggy.util.ZiggyShutdownHook.lambda$addShutdownHook$0(ZiggyShutdownHook.java:44) + at java.base/java.lang.Thread.run(Thread.java:833) + +____________________________________________________________________ +Job Resource Usage Summary for 20255559.pbspl1.nas.nasa.gov + + CPU Time Used : 02:30:11 + Real Memory Used : 19276372kb + Walltime Used : 00:30:29 + Exit Status : 271 + + Number of CPUs Requested : 1 + Walltime Requested : 00:30:00 + + Execution Queue : low + Charged To : s1007 + + Job Stopped : Wed Aug 7 12:30:55 2024 +____________________________________________________________________ diff --git a/test/data/PbsLogParser/pbs-log-status-no-comment.txt b/test/data/PbsLogParser/pbs-log-status-no-comment.txt new file mode 100644 index 0000000..0c4cea0 --- /dev/null +++ b/test/data/PbsLogParser/pbs-log-status-no-comment.txt @@ -0,0 +1,26 @@ +Job 20255559.pbspl1.nas.nasa.gov started on Wed Aug 07 12:00:12 PDT 2024 +The job requested the following resources: + ncpus=1 + place=scatter:excl + walltime=00:30:00 + +PBS set the following environment variables: + FORT_BUFFERED = 1 + TZ = PST8PDT + +____________________________________________________________________ +Job Resource Usage Summary for 20255559.pbspl1.nas.nasa.gov + + CPU Time Used : 02:30:11 + Real Memory Used : 19276372kb + Walltime Used : 00:30:29 + Exit Status : 0 + + Number of CPUs Requested : 1 + Walltime Requested : 00:30:00 + + Execution Queue : low + Charged To : s1007 + + Job Stopped : Wed Aug 7 12:30:55 2024 +____________________________________________________________________