Alternative proposal: pass a path to an executable instead of a version. This would leave the door open for supporting arbitrary executables in the future. So far nothing in the API is actually specific to Python or even a compiler/interpreter. The downside is that it's less user-friendly to know the full path to an executable than it is to specify a Python version.

So instead of discovering which versions are available, users could discover which binaries are available and what their paths are. We don't need to list everything that's mounted; can continue to use the same approach of configuring that in a JSON file.

This is an interesting idea, I'll into implementing it. I'd probably keep the display names in the JSON files so end-users (bot) can still display human-readable names.

Another approach would be to base it off nsjail config files. Each binary (including different Python versions) would have its own config file, and we would scan a directory for config files. The API would accept a config name. This would offer more flexibility in supporting multiple binaries, as different binaries may require different mounts, environment variables, etc. I don't think this approach is mutually exclusive with my previous proposal, so we can revisit this in the future as well.

I wonder how much of the config will be repeated. I imagine a lot of the python (for the cpython implementations) configs will be very repetitive, but even other interpreters would presumably need similar cgroup configuration, mounts, etc. We might be able to get around the duplication problem by providing overrides to be applied in python (or if nsjail supports combining configs?) instead of multiple config files. Either way, it's a problem for another day.

Yeah, fair enough if we stick with something like versions.json. However, I am considering that maybe arbitrary paths is better. It's more flexible, it's less complex, and requires no additional configuration. What problem or use case are we trying to solve by restricting the binaries to a predefined set?

We'd need to define the python versions statically somewhere at any rate, so we can actually define them in the dockerfile. We might be able to get away with an image that has all python versions already so we don't need to specify them, but that would require an external image most likely. I linked an example in the original issue, but that is no longer maintained.

On a more general level though, I'm not sure arbitrary binaries is the best approach. Different binaries would require different configurations, be that resources, mounts, or permissions. The most flexible system would be statically defined sets of configurations - harking back to your multiple configs idea - and at that point we'd end up with end users selecting from a pre-defined set of human names.

This system could actually define multiple binaries per config-set as well by passing the path as an argument, or more ideally, defining the paths in a dict and passing the name to the API which resolves the correct binary.

The most flexible system would be statically defined sets of configurations

I agree, and we want to leave the door open for that in the future.

Different binaries would require different configurations, be that resources, mounts, or permissions.

This is true, but not always. It is feasible for a single NsJail config to support multiple binaries (though it can only have one default entry point I think).

We'd need to define the python versions statically somewhere at any rate

That only applies to Python (so far), and I consider it an implementation detail as far as the API is concerned. I believe we need a stronger reason to influence the design of our API.

defining the paths in a dict and passing the name to the API which resolves the correct binary.

I find this redundant. It seems simpler and more intuitive to create an NsJail config, which effectively defines an environment, and have everything in that environment be available via the API.

It is feasible for a single NsJail config to support multiple binaries (though it can only have one default entry point I think).

This is true, but we can probably just avoid defining a default entry point in the first place. That's more or less what this PR does for python.

It seems simpler and more intuitive to create an NsJail config, which effectively defines an environment, and have everything in that environment be available via the API.

My problem is it moves the burden of knowing what binaries are available on the API from the project (which can reasonably control the matter) to the user (which can not directly affect it). For instance, if we use this PR to implement support for multiple python versions on bot, we can't trivially define what versions are available to the bot without essentially hardcoding it. A change to snekbox would necessitate a change to the users as well.

Regardless of which approach we choose, it'd be our burden as maintainers to ensure if we provide a binary for eval in one version, that we also have it available for future versions, or declaring an incompatibility if needed. This is much harder when we don't actually know what binaries users need, and have to assume everything is used.

Explicitly defining which binaries the API supports (and returning them in /info) alleviates both problem. At that point a display name is just the icing on the cake.

My problem is it moves the burden of knowing what binaries are available on the API from the project (which can reasonably control the matter) to the user (which can not directly affect it).

I don't necessarily see that as a problem. I've seen snekbox as primarily an internal service for which users build their own front ends. Thus, the users are in control of both, and they will be aware of what is available in the snekbox environment. It'd be a different story if the snekbox API was directly available on the internet for anyone to use, but I do not think that is the use case we should be prioritising.

This is much harder when we don't actually know what binaries users need, and have to assume everything is used.

We could include a list of binaries in our documentation, and it does not need to be exhaustive. We can just list the Python binaries. If users want to use other binaries, then that's on them, and we won't guarantee the availability of those undocumented binaries.