Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add sarif support for resolve function #63

Open
9 tasks
wandmagic opened this issue Oct 23, 2024 · 5 comments
Open
9 tasks

add sarif support for resolve function #63

wandmagic opened this issue Oct 23, 2024 · 5 comments
Labels
enhancement New feature or request

Comments

@wandmagic
Copy link

User Story: As a developer

I would like resolve functions to support optional SARIF format error output so that I can better track, analyze and integrate error reporting.

Goals:

Enable resolve functions to optionally output errors in SARIF (Static Analysis Results Interchange Format) format to provide standardized, detailed error reporting that can be consumed by various development tools.

Key requirements:

  • Add optional SARIF output format for error reporting in resolve functions
  • Support capturing runtime errors, validation errors, and other issues
  • Include relevant metadata like error location, severity, and error code
  • Allow integration with existing SARIF-compatible tools and CI/CD pipelines

Dependencies:

  • SARIF specification implementation library
  • Error handling middleware/utilities that can format errors into SARIF
  • Test infrastructure for validating SARIF output

Acceptance Criteria

  • All website and readme documentation affected by the changes in this issue have been updated
  • A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR
  • The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR
  • Resolve functions include an option to output errors in valid SARIF 2.1.0 format
  • Error output includes required SARIF fields (tool info, results, locations)
  • Runtime errors are properly captured and formatted in SARIF
  • Unit tests verify SARIF output structure and content
  • Documentation includes examples of SARIF error output and configuration
  • Performance impact of SARIF formatting is minimal and documented
@wandmagic wandmagic added the enhancement New feature or request label Oct 23, 2024
@aj-stein-gsa
Copy link
Contributor

@david-waltermire I had a conversation related to this request later in the afternoon with @wandmagic. Is it possible, given what I know about "the sandwich" of commands up through m-j, lo-j, and oscal-cli to validate the profile (I am not sure the full scope of the disabled feature, but it seems relevant per

IBoundLoader loader = OscalBindingContext.instance().newBoundLoader();
loader.disableFeature(DeserializationFeature.DESERIALIZE_VALIDATE_CONSTRAINTS);
Format asFormat;
// attempt to determine the format
if (cmdLine.hasOption(AS_OPTION)) {
try {
String asFormatText = cmdLine.getOptionValue(AS_OPTION);
asFormat = Format.valueOf(asFormatText.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException ex) {
return ExitCode.INVALID_ARGUMENTS
.exitMessage("Invalid '--as' argument. The format must be one of: " + Arrays.stream(Format.values())
.map(Enum::name)
.collect(CustomCollectors.joiningWithOxfordComma("or")));
}
} else {
// attempt to determine the format
try {
asFormat = loader.detectFormat(ObjectUtils.notNull(source));
} catch (FileNotFoundException ex) {
// this case was already checked for
return ExitCode.IO_ERROR.exitMessage("The provided source file '" + source + "' does not exist.");
} catch (IOException ex) {
return ExitCode.PROCESSING_ERROR.exit().withThrowable(ex);
} catch (IllegalArgumentException ex) {
return ExitCode.INVALID_ARGUMENTS.exitMessage(
"Source file has unrecognizable format. Use '--as' to specify the format. The format must be one of: "
+ Arrays.stream(Format.values())
.map(Enum::name)
.collect(CustomCollectors.joiningWithOxfordComma("or")));
}
}
) and opt to validate with inline STDOUT/STDERR output and optionally SARIF? Paul sees a need in conjunction with the server work as a result in the GSA/fedramp-automation#719.

Thoughts?

@david-waltermire
Copy link
Contributor

Yeah. We could add a --validate switch for this. In the mean time, you can always resolve then validate in separate commands. We could also get fancy and have --validate-source and --validate-output. We would need a switch for the SARIF output. Maybe --sarif or something similar?

@david-waltermire david-waltermire moved this from To Triage to Ready in Spec and Tooling Work Board Oct 23, 2024
@aj-stein-gsa
Copy link
Contributor

Yeah. We could add a --validate switch for this. In the mean time, you can always resolve then validate in separate commands. We could also get fancy and have --validate-source and --validate-output. We would need a switch for the SARIF output. Maybe --sarif or something similar?

@wandmagic sounds like Mikey likes it (to translate the TV show reference, Dave and I are Mikey 😆). I think we need to get through lower-level m-j release and update cycle and maybe this is up for grabs in a subsequent release then?

@wandmagic
Copy link
Author

I like --validate or --sarif with just a simple doc available check for the desired item, but it would be interesting if when we have an unhandled exception during execution we can get sarif logs of that error as well, this might be a different issue but same line of thought that got me here.

@david-waltermire
Copy link
Contributor

I like --validate or --sarif with just a simple doc available check for the desired item, but it would be interesting if when we have an unhandled exception during execution we can get sarif logs of that error as well, this might be a different issue but same line of thought that got me here.

@wandmagic Handling and exposing profile resolution errors via SARIF is very different. I'd like to have a higher bandwidth conversation with you about this to better understand the use case. We should summarize that conversation here.

@david-waltermire david-waltermire moved this from Ready to Backlog in Spec and Tooling Work Board Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

3 participants