checksums.json support clutters log with false crash messages

[Flamefire]

Since #3749 introduced support for checksums.json the log will now contain the following highly misleading error on almost every build:

ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in init): Couldn't find file checksums.json anywhere, and downloading it is disabled... Paths attempted (in order): ...

I recently got this from another admin who is debugging a failure and thought this is the first error, which is understandable given the message "EasyBuild crashed with an error" which is clearly wrong at this point: The failure is very much expected.
So EasyBuildError should not log anything on construction because it might be caught invalidating the logged message.

I found sys.excepthook which is called for unhandled exceptions which is likely a better place for logging the exception then logging every (such) exception on creation even if it is caught and handled as part of an expected code path.

Alternative: Let obtain_file return None in case the file was not found, maybe based on a parameter expect_success for backwards compatibility.

CC @mboisson

[mboisson]

🤔 Could you be more specific on which recipe crashes and what file it is attempting to download ? I doubt that the PR made almost every build fail (otherwise it would not have been merged)

[Flamefire]

Oh sorry for the confusion, the build is not crashing but the log contains the message "EasyBuild crashed with an error" because almost no ECs have a matching checksums.json but obtain_file doesn't (yet) care and raises an EasyBuildError which on construction logs that message.

[mboisson]

🤔 if that exception does not make it crash (because it is then handled by the caller), why is it logged at all ? The obtain_file method called with no_download = True should raise an exception if the file is not found without downloading it.

[Flamefire]

thinking if that exception does not make it crash (because it is then handled by the caller), why is it logged at all ?

Because EasyBuildError logs that message on construction. I.e. at that point it is not known if it is caught. This turns up in other situations too

[boegel]

@Flamefire The problem you're reporting is a very valid one, there should be no scary-looking " EasyBuild crashed with an error" messages in the middle of the log.

It's being fixed in #4261, by letting obtain_file only log a warning.

[Flamefire]

I see, so #4261 implements the alternative approach I suggested above.

However there are a dozen other instances where we catch this error (still logging the scary message) as identified by grep -rF "except EasyBuildError".
Some are even in easyblocks, e.g. openssl.py catches it and then logs an info message.

So maybe we want to change that log-on-construction logic anyway.

[boegel]

@Flamefire This was closed because the exact issue being reported here was fixed by #4261

Let's open a dedicated issue for coming up with a good way to prevent errors that should just be warnings from appearing in the log file.