Starting tagging functionality

[pearce8]

• Set of tags for benchpark: A central place to keep all the possible tag values.
• Need to be able to read tags in application.py, whether in benchpark/repo, or in the Ramble built-ins.
• Need functionality to report on the overall tags covered by experiments.
• Need functionality to report on the overall tags for each suite.
• When adding new tags to experiments, should check on PR that a tag exists, and encourage the users to pick from existing tags. If a new tag is needed, propose to add it to the benchpark_tags.yaml before starting to use it in experiments.

[scheibelp]

I think this should be automatically constructed from ramble attributes --tags --all. Note that this satisfies:

Need to be able to read tags in application.py, whether in benchpark/repo, or in the Ramble built-ins.

as long as the ramble config includes both of these repos.

Need functionality to report on the overall tags for each suite.

what is a suite? Is it just another tag that happens to be used for organization? For example I don't see a designated suite attribute for Ramble application objects.

If a new tag is needed, propose to add it to the benchpark_tags.yaml before starting to use it in experiments.

FWIW, one of our CI checks could be to auto-collect the tags and compare them with benchpark_tags.yaml, and fail automatically if they don't match. Then any PR would have to update benchpark_tags.yaml as part of the diff (we can likewise provide a command to do that update automatically.

[becker33]

Some tags seem to be per-application and some tags seem to be per-experiment (since applications can stress the system in different ways with different arguments. Should we check with the Ramble developers about adding a tagging capability to the ramble.yaml syntax.

These tags should then filter into benchpark list so that users can select tags for attributes of the system they want to test and get a list of potentially relevant experiments to choose from. Maybe benchpark list -t memory-bandwidth

Maybe we should publish (potentially outside of benchpark?) a taxonomy of benchmarking types which could be used as the set of available tags in benchpark? I'm not sure whether this should be coupled into benchpark or treated as an external dependency? I guess it's easier to start internal and spin out if necessary.

[becker33]

@scheibelp I think that we may want our list to be separate from the list of tags used in Ramble, because I think there are tags relevant to experiments that are not applicable at the application level.

I agree that we should have a CI check that all tags in our repo are in the list of tags, but I don't know that we want to necessarily care about every tag that ramble has in upstream packages. If someone tags an application as "google" because they developed it, we probably don't care from a benchpark perspective

[scheibelp]

I think that we may want our list to be separate from the list of tags used in Ramble, because I think there are tags relevant to experiments that are not applicable at the application level.

Agree, I think that was a possible reason for expanding the YAML format from a list of tags to a dict, which was collapsed in #65 (comment) (I should have made that a separate comment vs. an annotation).

I agree that we should have a CI check that all tags in our repo are in the list of tags, but I don't know that we want to necessarily care about every tag that ramble has in upstream packages. If someone tags an application as "google" because they developed it, we probably don't care from a benchpark perspective

Does this just mean that we might not want to display all tags that exist in all application.py's (and you think it makes sense to report a problem if a user mentions a tag that is not in an application.py, or possibly in a "group" as proposed above in this comment)? Without that grouping, I think tags outside of the set exported by applications aren't useful, unless you had some notion of another reason we could use them.

[alecbcs]

Looks good to me!