Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors in swebench.harness.run_evaluation in Windows #260

Closed
3N4N opened this issue Nov 26, 2024 · 0 comments · Fixed by #261
Closed

Errors in swebench.harness.run_evaluation in Windows #260

3N4N opened this issue Nov 26, 2024 · 0 comments · Fixed by #261
Labels
bug Something isn't working

Comments

@3N4N
Copy link
Contributor

3N4N commented Nov 26, 2024

Describe the bug

When I want to run the evaluation script on predictions already generated by a model, and I run the evaluation script swebench.harness.run_evaluation in a Windows system, it causes several issues. This is because SWE-bench is not completely OS independent.

Steps/Code to Reproduce

 python -m swebench.harness.run_evaluation --predictions_path output/predictions-from-model.jsonl --max_workers 1 --run_id validate-predictions

Expected Results

The evaluation runs without error.

Actual Results

Several errors occur.

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "C:\Users\enan\projects\SWE-bench\swebench\__init__.py", line 46, in <module>
    from swebench.harness.run_evaluation import (
  File "C:\Users\enan\projects\SWE-bench\swebench\harness\run_evaluation.py", line 5, in <module>
    import resource
ModuleNotFoundError: No module named 'resource'

This import error can be easily fixed just by commenting it out. But there are other OS-dependent issues that pops up.

System Information

WIndows 11, Python 3.12, swebench git+7501f0993193

@3N4N 3N4N added the bug Something isn't working label Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant