Comparing output to files (as opposed to stdout)

[niemela]

Let's say you wanted to define the following problem (and I would argue that something like this would be useful in an educational setting):

Read several numbers from numbers.txt, one number per line. Write their sum to sum.txt and their product to product.txt.

We can define this problem in the current version. You would put the numbers.txt-files as <testcase>.files/numbers.txt. For validating you would have to use a custom validator. You would put the sum and product in <testcase>.ans and the validator would get .ans as usual, and would have to know how to parse it and what should go in what file. This is not very different from any validator that gets some kind of secret from the .ans file.

The implication is that you will always have to use a custom validator when you want the submission to write output to a file.

If we don't want this we could introduce something like an .ansfiles directory. The default validator would then for each file in .ansfiles verify that this exists and is equal in the working directory of the submission after running it.

My sense is that requiring a custom validator whenever you use file output is acceptable, and that adding something like .ansfiles is more complication than it's worth. So, I suggest that we don't do anything. I think @Tagl also agrees?

Thoughts?

---

A related question is allowing writing at all.

I would argue that we do want the kind of problems suggested above, and then we have to allow writing to file (that is in fact the point). That said, in the normal stdin/stdout case it would be nice if the system is a at least allowed to forbid writing to files. Should there be a setting that you are required (as a problem setter) to use to explicitly allow file writes? This would also allow a system to know at install time that a problem will require writing to file, and thus be able to refuse the problem at install time, rather than failing at run time.

[Tagl]

I think @Tagl also agrees?

I do agree.

Should there be a setting that you are required (as a problem setter) to use to explicitly allow file writes?

I think yes, boolean is probably enough. Potentially you would want to have an additional setting that says writing new files is allowed but editing submitted files is not, or even 4 settings if you consider included files too. I think that's not needed though and results in too much complexity without a proper use case in mind.

[eldering]

Thinking about our setup in DOMjudge, if allowing submissions to write to files, I'd prefer if the problem would then somehow specify which files and/or directories, or to have a default directory to write to, like the feedback_dir for validators. That way, I can shield off anything else by not giving the submission write permission to any file or directory than those explicitly allowed.

[evouga]

My sense is that requiring a custom validator whenever you use file output is acceptable, and that adding something like .ansfiles is more complication than it's worth.

Agreed as well.

I'd prefer if the problem would then somehow specify which files and/or directories, or to have a default directory to write to

Under the current spec this will be the submission's working directory. Is that OK?

[niemela]

It sounds like we all agree on the first part. 👍

---

I'd prefer if the problem would then somehow specify which files and/or directories, or to have a default directory to write to

Under the current spec this will be the submission's working directory. Is that OK?

IMO it really must be that.

Should there be a setting that you are required (as a problem setter) to use to explicitly allow file writes?

I think yes, boolean is probably enough.

So something like "if true, you can create and edit files"?

Potentially you would want to have an additional setting that says writing new files is allowed but editing submitted files is not,

By "submitted files" I assume you mean by the user. What is the use case where this would be useful? I don't like the implication that a submission could rewrite itself between multi-pass rounds, or in the case of more dynamic languages, even during the current run.

or even 4 settings if you consider included files too.

How does that become 4? Isn't it "allow create", "allow edit submitted", "allow edit included"?

Of course there is also the test case files. Is "allow edit test case files" the 4th one?

I think that's not needed though and results in too much complexity without a proper use case in mind.

Yes, I agree.

[Tagl]

How does that become 4?

One option is not allowing file editing at all

[niemela]

How does that become 4?

One option is not allowing file editing at all

Right, but I count 3 settings (or choices), which is 8 options. Which 4 are you counting?

[Tagl]

Right, when I wrote this I just thought in levels of permissiveness 0-3 essentially, not multiple binary flags. But binary flags would be more expressive. Again, at this time I think these options are not worth the effort and a single boolean flag is all we need.