Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

probes: fileformats probe fails when targeting a local modal path #1034

Open
jmartin-tech opened this issue Dec 3, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@jmartin-tech
Copy link
Collaborator

The fileformats are limited to huggingface generators, however there are cases where the generator can load a local checkpoint that would not be complatible with this probe. This is due the the expectation that the model under test is published for file enumeration via the huggingface_hub utility library.

Contrived example:

python -m garak -m huffingface.Model -n /usr/local/checkpoints/Mistral-NeMo-Minitron-8B-Instruct -p fileformats

Current trace reported in logs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/garak/cli.py", line 530, in main
    command.probewise_run(
  File "/usr/local/lib/python3.10/site-packages/garak/command.py", line 237, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/usr/local/lib/python3.10/site-packages/garak/harnesses/probewise.py", line 107, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/usr/local/lib/python3.10/site-packages/garak/harnesses/base.py", line 114, in run
    attempt_results = probe.probe(model)
  File "/usr/local/lib/python3.10/site-packages/garak/probes/fileformats.py", line 63, in probe
    repo_filenames = huggingface_hub.list_repo_files(generator.name)
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/usr/local/checkpoints/Mistral-NeMo-Minitron-8B-Instruct/'. Use `repo_type` argument if needed.

This probe needs to be able to either skip for local model paths or follow an alternative file gathering approach.

@jmartin-tech jmartin-tech added the bug Something isn't working label Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant