What decides whether a dependency is part of the "instance" or "environment" level cache?

[max-kaufmann]

Describe the issue

I've implemented a version of SWE-bench in InspectAI (here), and we use cache level = "environment".

We're a bit confused what the reasoning is between where things are cached. Looking at the implementation, it seems that its based on the various fields in MAP_REPO_VERSION_TO_SPECS. It seems that:

1. enviornment images pip install the package list in "pip packages" and installs from
2. Install a set of dependencies defined by the "packages" field (which can be a dependency file, or a list of python packages).

The repo_setup_script (which is per-instance) will then run

For example for SKLEARN:

https://github.com/princeton-nlp/SWE-bench/blob/b31927c0c9dd306c638ab10fe0febdff45ecfa7e/swebench/harness/constants.py#L60-L73

It would pip install "numpy scipy cython pytest pandas matplotlib", and also pip install "cython, numpy==1.19.2, setuptools, scipy==1.5.2".

The repo_setup script (which is ran per instance, not per environment), will run the "pre_install" and "install" commands. Why is this the case? It seems you're already installing from the list of dependencies in the repo in the "env" step.

Suggest an improvement to documentation

I think that more writing/descriptions of why it is architected how it is would be really helpful for building on your codebase!