Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement] Various enhancements to the results DB feature #3283

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

vkarak
Copy link
Contributor

@vkarak vkarak commented Oct 11, 2024

More specifically, this PR adds the following enhancements:

  • Support for computed virtual test attributes. This mechanism allows us to extend the test queries in existing databases without having to migrate the DBs and update the test records.
    • basename: the test's name without any parameters (essentially the test class name). This is useful for parameter-based aggregations.
    • sysenv: the formatted system, partition, environment combination
    • pdiff: the performance difference when doing comparisons
  • Support for completely custom groupings. The syntax os the CMPSPEC is extended to get an explicit list of attributes for grouping test cases.
  • Support for arbitrary table column selection. The syntax of the CMPSPEC is extended to get an explicit list of attribute columns to show. As a result the --table-hide-columns option is removed as now the exact columns of the output can be specified.
  • Extend --list-stored-testcases to show the performance summary of the selected cases. It uses the same CMPSPEC syntax.
  • Format parameters when storing a report/session using the parameter's format function.
  • Show A/B absolute values for non-aggregated attributes/columns.
  • Enable -E for testcase filtering.
  • Session extras keys are now stored unchanged in the DB (previously, they were prefixed with $). This allows them to be properly filtered with the session filtering expressions passed to --session-filter.
  • The --session-extras option can now be specified multiple times.
  • Replace pretty table format with outline and add a new grid format. This is to allow nice printing of multiline cells.
  • Remove --session-filter option and replace it with special syntax in the CMPSPEC. This allows us to select and compare different sets of sessions.
  • Clean up and homogenize the internal storage API.
  • All options dealing with sessions (e.g., --list-stored-sessions, --delete-sessions) are now accepting the extended query syntax (over time period, session uuid or session filter).
  • Options that used to manipulate a single session have now their names converted to plural.

Docs and unit tests are updated accordingly.

@vkarak vkarak added prio: normal enhancement reporting Issues related to reporting and processing the test results labels Oct 11, 2024
@vkarak vkarak added this to the ReFrame 4.7 milestone Oct 11, 2024
@vkarak vkarak requested review from ekouts and teojgo October 11, 2024 11:54
@vkarak vkarak self-assigned this Oct 11, 2024
- Support for computed virtual test attributes
  - `basename`: the test's name without any parameters
  - `sysenv`: the formatted system, partition, environment combination
  - `pdiff`: the performance difference when doing comparisons
- Support for completely custom groupings
- Support for arbitrary table column selection
- Extend `--list-stored-testcases` to show the performance summary of
  the selected cases
- Format parameters when storing a report/session using the parameter's
  format function
- Show A/B absolute values for non-aggregated attributes/columns
@vkarak vkarak force-pushed the feat/results-db-enhancements branch from b708cc6 to eafcdeb Compare October 11, 2024 12:08
@vkarak
Copy link
Contributor Author

vkarak commented Oct 11, 2024

As another improvement, what would you think of making --session-extras an "append" option so that it can be specified multiple times to augment any "default" extras passed for example in a pipeline?

@vkarak vkarak requested a review from victorusu October 18, 2024 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement prio: normal reporting Issues related to reporting and processing the test results
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

1 participant