Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reporting: enable logging & skipping of unrecognised probes, detectors in digest generation #1024

Open
leondz opened this issue Nov 25, 2024 · 0 comments
Labels
reporting Reporting, analysis, and other per-run result functions

Comments

@leondz
Copy link
Collaborator

leondz commented Nov 25, 2024

Summary

If an unknown probe or detector is encountered in report_digest, due for example to version mismatches, it fails. Would be cool if it didn't.

Basic example

Here, the plugin name's shifted from ContinueSlursReclaimedSlurs80 to ContinueSlursReclaimedSlursMini`:

(garak) 10:04:50 x1:~/dev/garak [main] $ python -m garak.analyze.report_digest hymba.jsonl > hymba.html
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/lderczynski/dev/garak/garak/analyze/report_digest.py", line 292, in <module>
    digest_content = compile_digest(report_path, taxonomy=taxonomy)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lderczynski/dev/garak/garak/analyze/report_digest.py", line 201, in compile_digest
    getattr(pm, probe_class).__doc__
    ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'garak.probes.continuation' has no attribute 'ContinueSlursReclaimedSlurs80'. Did you mean: 'ContinueSlursReclaimedSlurs'?

Would be cool if we notified on stderr and in the report, and continued on, instead of breaking like this.

Motivation

To enable reports to be generated even if not all the source data can be processed as we'd like

@leondz leondz added the reporting Reporting, analysis, and other per-run result functions label Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
reporting Reporting, analysis, and other per-run result functions
Projects
None yet
Development

No branches or pull requests

1 participant